hobbs-iac/docs/plans/2026-03-07-frigate-notify-v2-plan.md

Frigate-Notify v2 Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Evolve frigate-notify into an intelligent notification system with known-object recognition (CLIP + Vision LLM), a web UI for managing identities and reviewing events, and configurable vision backends (local Ollama or Claude Haiku).

Architecture: A standalone Python/FastAPI web app (:5199) that listens for Frigate MQTT events, matches snapshots against a known-objects database using CLIP embeddings (via a GPU sidecar on 192.168.86.196), falls back to vision LLM for ambiguous matches, and only sends Matrix alerts for unknown objects. A vanilla HTML/CSS/JS web UI provides event review, object tagging, and settings management.

Tech Stack: Python 3.12, FastAPI, SQLite, paho-mqtt, aiohttp, open_clip (sidecar), Ollama, Claude Haiku API, vanilla HTML/CSS/JS

Design Doc: docs/plans/2026-03-07-frigate-notify-v2-design.md

---

Context for the Implementer

Existing Code

The v1 app lives at /mnt/docker-data/compose/hf-pdocker-01/frigate-notify/ on the NAS and in the Gitea repo jeremy/frigate-notify. The source is also in the local repo at /mnt/c/Users/Jeremy/projects/docker/compose/hf-pdocker-01/frigate-notify/.

frigate-notify/
  docker-compose.yml      # Compose definition
  Dockerfile              # python:3.12-slim based
  .env                    # ANTHROPIC_API_KEY, MATRIX_ACCESS_TOKEN, MATRIX_ROOM_ID
  app/
    main.py               # MQTT listener, event processor, health endpoint (~200 lines)
    vision.py             # Claude Haiku vision wrapper (~30 lines)
    matrix_client.py      # Matrix API client (~80 lines)
    requirements.txt      # paho-mqtt, anthropic, aiohttp

Key Services

Service	Host	Port	Purpose
Frigate NVR	192.168.86.192 (hf-pdocker-01)	5000	Camera NVR, snapshots API
Mosquitto	192.168.86.192	1883	MQTT broker
Matrix/Synapse	192.168.86.167 (bart, failover)	8008	Chat server
Ollama	192.168.86.196	11434	Local LLM inference (RTX 3090)
Farm Manager	varies	8888	Dashboard (sidebar link target)

Git Repo

• Gitea: http://192.168.86.100:3003/jeremy/frigate-notify
• Clone URL: http://jeremy:806153c545dc3b227ecdb86b134a2e50210a8e89@192.168.86.100:3003/jeremy/frigate-notify.git
• Local working copy: /mnt/c/Users/Jeremy/projects/frigate-notify/ (create if doesn't exist)

SSH Access (for deployment)

• Username: glyptic, Password: &Rip8a115
• Use: sshpass -p '&Rip8a115' ssh -o StrictHostKeyChecking=no glyptic@<IP>

Note on hf-pdocker-01

hf-pdocker-01 is currently DOWN (hard drive failure). Frigate-notify can't run until it's restored (it needs Frigate + MQTT on that node). Development and testing should be done locally and via the Gitea repo. The clip-service sidecar on 192.168.86.196 CAN be deployed and tested independently.

---

Task 1: Project Scaffolding & FastAPI Migration

Goal: Restructure the project for v2 — migrate from raw HTTP to FastAPI, add proper project layout with tests, set up SQLite database.

Files:

• Modify: app/main.py (replace HTTPServer with FastAPI)
• Create: app/database.py (SQLite schema and connection)
• Create: app/models.py (Pydantic models)
• Create: app/config.py (centralized configuration)
• Modify: app/requirements.txt (add fastapi, uvicorn, python-multipart)
• Create: tests/conftest.py
• Create: tests/test_database.py
• Modify: Dockerfile (change CMD to uvicorn)

Step 1: Clone the repo locally and set up project structure

cd /mnt/c/Users/Jeremy/projects
git clone http://jeremy:806153c545dc3b227ecdb86b134a2e50210a8e89@192.168.86.100:3003/jeremy/frigate-notify.git
cd frigate-notify

Step 2: Create app/config.py

import os

class Config:
    # MQTT
    MQTT_HOST: str = os.environ.get("MQTT_HOST", "192.168.86.192")
    MQTT_PORT: int = int(os.environ.get("MQTT_PORT", "1883"))

    # Frigate
    FRIGATE_URL: str = os.environ.get("FRIGATE_URL", "http://192.168.86.192:5000")

    # Matrix
    MATRIX_HOMESERVER: str = os.environ.get("MATRIX_HOMESERVER", "http://192.168.86.192:8008")
    MATRIX_ACCESS_TOKEN: str = os.environ.get("MATRIX_ACCESS_TOKEN", "")
    MATRIX_ROOM_ID: str = os.environ.get("MATRIX_ROOM_ID", "")

    # Vision backends
    CLIP_SERVICE_URL: str = os.environ.get("CLIP_SERVICE_URL", "http://192.168.86.196:5200")
    OLLAMA_URL: str = os.environ.get("OLLAMA_URL", "http://192.168.86.196:11434")
    OLLAMA_VISION_MODEL: str = os.environ.get("OLLAMA_VISION_MODEL", "llava:13b")

    # Recognition thresholds
    MATCH_HIGH_CONFIDENCE: float = float(os.environ.get("MATCH_HIGH_CONFIDENCE", "0.85"))
    MATCH_LOW_CONFIDENCE: float = float(os.environ.get("MATCH_LOW_CONFIDENCE", "0.60"))

    # Filtering
    ALLOWED_LABELS: set = {"person", "car", "dog", "cat", "bird"}
    MIN_SCORE: float = 0.7
    RATE_LIMIT_SECONDS: int = 60
    DEDUP_WINDOW_SECONDS: int = 30

    # Server
    HOST: str = "0.0.0.0"
    PORT: int = 5199

    # Database
    DB_PATH: str = os.environ.get("DB_PATH", "/data/frigate-notify.db")

config = Config()

Step 3: Create app/database.py

import sqlite3
import threading
from contextlib import contextmanager

_local = threading.local()


def get_connection(db_path: str) -> sqlite3.Connection:
    if not hasattr(_local, "connection") or _local.connection is None:
        _local.connection = sqlite3.connect(db_path, check_same_thread=False)
        _local.connection.row_factory = sqlite3.Row
        _local.connection.execute("PRAGMA journal_mode=WAL")
        _local.connection.execute("PRAGMA foreign_keys=ON")
    return _local.connection


def init_db(db_path: str) -> None:
    conn = sqlite3.connect(db_path)
    conn.executescript("""
        CREATE TABLE IF NOT EXISTS known_objects (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            name TEXT NOT NULL,
            type TEXT NOT NULL CHECK(type IN ('person', 'vehicle', 'pet')),
            action TEXT NOT NULL DEFAULT 'quiet' CHECK(action IN ('quiet', 'suppress', 'alert')),
            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
            updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );

        CREATE TABLE IF NOT EXISTS embeddings (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            known_object_id INTEGER NOT NULL REFERENCES known_objects(id) ON DELETE CASCADE,
            embedding BLOB NOT NULL,
            snapshot_url TEXT,
            thumbnail BLOB,
            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );

        CREATE TABLE IF NOT EXISTS events (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            frigate_event_id TEXT UNIQUE NOT NULL,
            camera TEXT NOT NULL,
            label TEXT NOT NULL,
            score REAL NOT NULL,
            snapshot BLOB,
            embedding BLOB,
            match_status TEXT NOT NULL DEFAULT 'unreviewed'
                CHECK(match_status IN ('known', 'unknown', 'ambiguous', 'unreviewed')),
            matched_object_id INTEGER REFERENCES known_objects(id) ON DELETE SET NULL,
            match_confidence REAL,
            ai_description TEXT,
            notified INTEGER NOT NULL DEFAULT 0,
            created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
        );

        CREATE TABLE IF NOT EXISTS settings (
            key TEXT PRIMARY KEY,
            value TEXT NOT NULL
        );

        CREATE INDEX IF NOT EXISTS idx_events_created ON events(created_at);
        CREATE INDEX IF NOT EXISTS idx_events_status ON events(match_status);
        CREATE INDEX IF NOT EXISTS idx_events_camera ON events(camera);
        CREATE INDEX IF NOT EXISTS idx_embeddings_object ON embeddings(known_object_id);
    """)
    # Insert default settings if not present
    defaults = {
        "vision_backend": "haiku",
        "recognition_enabled": "false",
        "event_retention_days": "90",
    }
    for key, value in defaults.items():
        conn.execute(
            "INSERT OR IGNORE INTO settings (key, value) VALUES (?, ?)",
            (key, value),
        )
    conn.commit()
    conn.close()

Step 4: Create app/models.py

from pydantic import BaseModel
from typing import Optional
from datetime import datetime


class KnownObjectCreate(BaseModel):
    name: str
    type: str  # person, vehicle, pet
    action: str = "quiet"  # quiet, suppress, alert


class KnownObjectUpdate(BaseModel):
    name: Optional[str] = None
    type: Optional[str] = None
    action: Optional[str] = None


class KnownObjectResponse(BaseModel):
    id: int
    name: str
    type: str
    action: str
    created_at: str
    updated_at: str
    embedding_count: int = 0


class TagEventRequest(BaseModel):
    known_object_id: Optional[int] = None
    name: Optional[str] = None
    type: Optional[str] = None


class EventResponse(BaseModel):
    id: int
    frigate_event_id: str
    camera: str
    label: str
    score: float
    match_status: str
    matched_object_id: Optional[int] = None
    matched_object_name: Optional[str] = None
    match_confidence: Optional[float] = None
    ai_description: Optional[str] = None
    notified: bool
    created_at: str


class SettingsResponse(BaseModel):
    vision_backend: str = "haiku"
    recognition_enabled: bool = False
    event_retention_days: int = 90
    match_high_confidence: float = 0.85
    match_low_confidence: float = 0.60
    rate_limit_seconds: int = 60
    allowed_labels: list[str] = ["person", "car", "dog", "cat", "bird"]
    min_score: float = 0.7


class StatsResponse(BaseModel):
    total_events: int
    known_matches: int
    unknown_events: int
    unreviewed: int
    known_objects_count: int
    events_today: int

Step 5: Update app/requirements.txt

paho-mqtt==2.1.0
anthropic>=0.40.0
aiohttp>=3.9.0
fastapi>=0.115.0
uvicorn[standard]>=0.34.0
python-multipart>=0.0.18
numpy>=1.26.0

Step 6: Rewrite app/main.py for FastAPI

This is the big refactor. The new main.py sets up FastAPI, mounts the MQTT listener as a background task, and serves both the API and static files.

import asyncio
import json
import logging
import os
import signal
import sys
import time
from collections import defaultdict
from threading import Thread

import aiohttp
import anthropic
import paho.mqtt.client as mqtt
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from fastapi.responses import FileResponse

from config import config
from database import init_db, get_connection
from vision import analyze_snapshot
from matrix_client import MatrixClient

# --- Logging ---
logging.basicConfig(
    level=logging.INFO,
    format='{"time":"%(asctime)s","level":"%(levelname)s","module":"%(module)s","message":"%(message)s"}',
    stream=sys.stdout,
)
logger = logging.getLogger(__name__)

# --- State ---
recent_events: dict[str, float] = {}
last_notify_per_camera: defaultdict[str, float] = defaultdict(float)
events_processed = 0
last_event_time: str | None = None
mqtt_connected = False

# --- App ---
app = FastAPI(title="Frigate Notify", version="2.0.0")

# Mount static files for web UI
static_dir = os.path.join(os.path.dirname(__file__), "static")
if os.path.isdir(static_dir):
    app.mount("/static", StaticFiles(directory=static_dir), name="static")


@app.on_event("startup")
async def startup():
    init_db(config.DB_PATH)
    start_mqtt()
    logger.info("Frigate-notify v2 started")


@app.get("/health")
async def health():
    return {
        "status": "ok",
        "version": "2.0.0",
        "mqtt_connected": mqtt_connected,
        "events_processed": events_processed,
        "last_event": last_event_time,
    }


@app.get("/")
async def index():
    return FileResponse(os.path.join(static_dir, "index.html"))


# --- Snapshot fetching ---
async def fetch_snapshot(event_id: str) -> bytes | None:
    url = f"{config.FRIGATE_URL}/api/events/{event_id}/snapshot.jpg"
    try:
        async with aiohttp.ClientSession() as session:
            async with session.get(url) as resp:
                if resp.status == 200:
                    data = await resp.read()
                    logger.info("Fetched snapshot for %s: %d bytes", event_id, len(data))
                    return data
                logger.warning("Snapshot fetch failed for %s: HTTP %d", event_id, resp.status)
    except Exception:
        logger.exception("Snapshot fetch error for %s", event_id)
    return None


# --- Event processing ---
async def process_event(event_data: dict, claude: anthropic.Anthropic, matrix: MatrixClient):
    global events_processed, last_event_time

    event_type = event_data.get("type")
    if event_type != "end":
        return

    after = event_data.get("after", {})
    event_id = after.get("id", "")
    camera = after.get("camera", "unknown")
    label = after.get("label", "")
    score = after.get("top_score", after.get("score", 0))
    has_snapshot = after.get("has_snapshot", False)

    if label not in config.ALLOWED_LABELS:
        return
    if score < config.MIN_SCORE:
        return
    if not has_snapshot:
        return

    now = time.time()
    if event_id in recent_events and now - recent_events[event_id] < config.DEDUP_WINDOW_SECONDS:
        return
    recent_events[event_id] = now

    if now - last_notify_per_camera[camera] < config.RATE_LIMIT_SECONDS:
        logger.info("Rate limited: %s on %s", label, camera)
        return
    last_notify_per_camera[camera] = now

    cutoff = now - config.DEDUP_WINDOW_SECONDS * 2
    for eid in [k for k, v in recent_events.items() if v < cutoff]:
        del recent_events[eid]

    logger.info("Processing event: %s on %s (%s, score=%.2f)", event_id, camera, label, score)

    snapshot = await fetch_snapshot(event_id)

    # Store event in database
    snapshot_thumb = None
    if snapshot:
        # Store a smaller thumbnail (first 100KB max)
        snapshot_thumb = snapshot[:102400] if len(snapshot) > 102400 else snapshot

    conn = get_connection(config.DB_PATH)
    try:
        conn.execute(
            """INSERT OR IGNORE INTO events
               (frigate_event_id, camera, label, score, snapshot, match_status)
               VALUES (?, ?, ?, ?, ?, 'unreviewed')""",
            (event_id, camera, label, score, snapshot_thumb),
        )
        conn.commit()
    except Exception:
        logger.exception("Failed to store event %s", event_id)

    # AI description
    description = None
    if snapshot:
        description = analyze_snapshot(claude, snapshot)
    if not description:
        description = f"{label.capitalize()} detected on {camera}."

    # Update event with description
    try:
        conn.execute(
            "UPDATE events SET ai_description = ? WHERE frigate_event_id = ?",
            (description, event_id),
        )
        conn.commit()
    except Exception:
        logger.exception("Failed to update event description")

    # Format and send Matrix notification
    score_pct = int(score * 100)
    ts = time.strftime("%I:%M:%S %p", time.localtime())
    date_str = time.strftime("%b %d, %Y", time.localtime())
    frigate_link = f"{config.FRIGATE_URL}/review?id={event_id}"

    plain = (
        f"{camera} - {label.capitalize()} - {score_pct}%\n"
        f"{description}\n\n"
        f"View in Frigate: {frigate_link}\n"
        f"{ts} - {date_str}"
    )
    html = (
        f"<b>{camera}</b> &middot; {label.capitalize()} &middot; {score_pct}%<br/>"
        f"{description}<br/><br/>"
        f'<a href="{frigate_link}">View in Frigate</a><br/>'
        f"<small>{ts} &middot; {date_str}</small>"
    )

    if snapshot:
        filename = f"{camera}_{event_id}.jpg"
        mxc_uri = await matrix.upload_image(snapshot, filename)
        if mxc_uri:
            await matrix.send_image(mxc_uri, filename, len(snapshot))

    await matrix.send_text(plain, html)

    # Mark as notified
    try:
        conn.execute(
            "UPDATE events SET notified = 1 WHERE frigate_event_id = ?",
            (event_id,),
        )
        conn.commit()
    except Exception:
        pass

    events_processed += 1
    last_event_time = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
    logger.info("Notification sent for %s on %s", label, camera)


# --- MQTT ---
def start_mqtt():
    global mqtt_connected

    claude = anthropic.Anthropic()
    matrix = MatrixClient(config.MATRIX_HOMESERVER, config.MATRIX_ACCESS_TOKEN, config.MATRIX_ROOM_ID)
    loop = asyncio.get_event_loop()

    def on_connect(client, userdata, flags, reason_code, properties):
        global mqtt_connected
        mqtt_connected = True
        client.subscribe("frigate/events")
        logger.info("MQTT connected, subscribed to frigate/events")

    def on_disconnect(client, userdata, flags, reason_code, properties):
        global mqtt_connected
        mqtt_connected = False
        logger.warning("MQTT disconnected: %s", reason_code)

    def on_message(client, userdata, msg):
        try:
            payload = json.loads(msg.payload)
            asyncio.run_coroutine_threadsafe(process_event(payload, claude, matrix), loop)
        except json.JSONDecodeError:
            logger.warning("Invalid JSON on %s", msg.topic)
        except Exception:
            logger.exception("Error processing MQTT message")

    mqtt_client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
    mqtt_client.on_connect = on_connect
    mqtt_client.on_disconnect = on_disconnect
    mqtt_client.on_message = on_message

    def mqtt_loop():
        try:
            mqtt_client.connect(config.MQTT_HOST, config.MQTT_PORT)
            mqtt_client.loop_forever()
        except Exception:
            logger.exception("MQTT connection failed")

    thread = Thread(target=mqtt_loop, daemon=True)
    thread.start()


# --- Run ---
if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host=config.HOST, port=config.PORT, log_level="info")

Step 7: Update Dockerfile

FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*

COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ .

RUN mkdir -p /data

EXPOSE 5199

CMD ["python", "-u", "main.py"]

Step 8: Create tests/conftest.py

import os
import sys
import tempfile
import pytest

# Add app directory to path
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "app"))


@pytest.fixture
def db_path():
    with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as f:
        path = f.name
    yield path
    os.unlink(path)

Step 9: Create tests/test_database.py

from database import init_db, get_connection


def test_init_db_creates_tables(db_path):
    init_db(db_path)
    conn = get_connection(db_path)
    tables = conn.execute(
        "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
    ).fetchall()
    table_names = [t["name"] for t in tables]
    assert "known_objects" in table_names
    assert "embeddings" in table_names
    assert "events" in table_names
    assert "settings" in table_names


def test_default_settings_inserted(db_path):
    init_db(db_path)
    conn = get_connection(db_path)
    row = conn.execute("SELECT value FROM settings WHERE key='vision_backend'").fetchone()
    assert row["value"] == "haiku"


def test_known_object_crud(db_path):
    init_db(db_path)
    conn = get_connection(db_path)
    conn.execute(
        "INSERT INTO known_objects (name, type) VALUES (?, ?)",
        ("Jeremy", "person"),
    )
    conn.commit()
    row = conn.execute("SELECT * FROM known_objects WHERE name='Jeremy'").fetchone()
    assert row["type"] == "person"
    assert row["action"] == "quiet"


def test_event_insert(db_path):
    init_db(db_path)
    conn = get_connection(db_path)
    conn.execute(
        """INSERT INTO events (frigate_event_id, camera, label, score, match_status)
           VALUES (?, ?, ?, ?, ?)""",
        ("evt-123", "back_porch", "person", 0.92, "unreviewed"),
    )
    conn.commit()
    row = conn.execute("SELECT * FROM events WHERE frigate_event_id='evt-123'").fetchone()
    assert row["camera"] == "back_porch"
    assert row["score"] == 0.92


def test_embedding_cascade_delete(db_path):
    init_db(db_path)
    conn = get_connection(db_path)
    conn.execute("INSERT INTO known_objects (name, type) VALUES ('Dog', 'pet')")
    obj_id = conn.execute("SELECT last_insert_rowid()").fetchone()[0]
    conn.execute(
        "INSERT INTO embeddings (known_object_id, embedding) VALUES (?, ?)",
        (obj_id, b"fake-embedding"),
    )
    conn.commit()
    conn.execute("DELETE FROM known_objects WHERE id=?", (obj_id,))
    conn.commit()
    rows = conn.execute("SELECT * FROM embeddings WHERE known_object_id=?", (obj_id,)).fetchall()
    assert len(rows) == 0

Step 10: Run tests

cd /mnt/c/Users/Jeremy/projects/frigate-notify
pip install pytest fastapi uvicorn python-multipart numpy paho-mqtt anthropic aiohttp
pytest tests/ -v

Expected: All 5 tests pass.

Step 11: Commit

git add -A
git commit -m "feat: scaffold v2 with FastAPI, SQLite database, and project structure"
git push origin main

---

Task 2: REST API Endpoints

Goal: Implement all CRUD API endpoints for events, known objects, and settings.

Files:

• Create: app/routes/__init__.py
• Create: app/routes/events.py
• Create: app/routes/known_objects.py
• Create: app/routes/settings.py
• Modify: app/main.py (include routers)
• Create: tests/test_api.py

Step 1: Create app/routes/__init__.py

# Routes package

Step 2: Create app/routes/events.py

import base64
from fastapi import APIRouter, Query, HTTPException
from database import get_connection
from config import config
from models import EventResponse, TagEventRequest

router = APIRouter(prefix="/api/events", tags=["events"])


@router.get("")
async def list_events(
    camera: str | None = None,
    label: str | None = None,
    status: str | None = None,
    limit: int = Query(default=50, le=200),
    offset: int = 0,
):
    conn = get_connection(config.DB_PATH)
    query = """
        SELECT e.*, ko.name as matched_object_name
        FROM events e
        LEFT JOIN known_objects ko ON e.matched_object_id = ko.id
        WHERE 1=1
    """
    params = []
    if camera:
        query += " AND e.camera = ?"
        params.append(camera)
    if label:
        query += " AND e.label = ?"
        params.append(label)
    if status:
        query += " AND e.match_status = ?"
        params.append(status)
    query += " ORDER BY e.created_at DESC LIMIT ? OFFSET ?"
    params.extend([limit, offset])

    rows = conn.execute(query, params).fetchall()
    return [
        {
            **{k: row[k] for k in row.keys() if k != "snapshot" and k != "embedding"},
            "has_snapshot": row["snapshot"] is not None,
            "notified": bool(row["notified"]),
        }
        for row in rows
    ]


@router.get("/{event_id}")
async def get_event(event_id: int):
    conn = get_connection(config.DB_PATH)
    row = conn.execute(
        """SELECT e.*, ko.name as matched_object_name
           FROM events e
           LEFT JOIN known_objects ko ON e.matched_object_id = ko.id
           WHERE e.id = ?""",
        (event_id,),
    ).fetchone()
    if not row:
        raise HTTPException(404, "Event not found")
    result = {k: row[k] for k in row.keys() if k != "embedding"}
    if row["snapshot"]:
        result["snapshot"] = base64.b64encode(row["snapshot"]).decode()
    else:
        result["snapshot"] = None
    result["notified"] = bool(row["notified"])
    return result


@router.get("/{event_id}/snapshot")
async def get_event_snapshot(event_id: int):
    from fastapi.responses import Response

    conn = get_connection(config.DB_PATH)
    row = conn.execute("SELECT snapshot FROM events WHERE id = ?", (event_id,)).fetchone()
    if not row or not row["snapshot"]:
        raise HTTPException(404, "Snapshot not found")
    return Response(content=row["snapshot"], media_type="image/jpeg")


@router.post("/{event_id}/tag")
async def tag_event(event_id: int, body: TagEventRequest):
    conn = get_connection(config.DB_PATH)
    event = conn.execute("SELECT * FROM events WHERE id = ?", (event_id,)).fetchone()
    if not event:
        raise HTTPException(404, "Event not found")

    if body.known_object_id:
        obj = conn.execute(
            "SELECT * FROM known_objects WHERE id = ?", (body.known_object_id,)
        ).fetchone()
        if not obj:
            raise HTTPException(404, "Known object not found")
        object_id = body.known_object_id
    elif body.name and body.type:
        cursor = conn.execute(
            "INSERT INTO known_objects (name, type) VALUES (?, ?)",
            (body.name, body.type),
        )
        object_id = cursor.lastrowid
    else:
        raise HTTPException(400, "Provide known_object_id or name+type")

    # Store embedding if event has one
    if event["embedding"]:
        conn.execute(
            """INSERT INTO embeddings (known_object_id, embedding, thumbnail)
               VALUES (?, ?, ?)""",
            (object_id, event["embedding"], event["snapshot"]),
        )

    conn.execute(
        """UPDATE events SET match_status = 'known', matched_object_id = ?,
           match_confidence = 1.0 WHERE id = ?""",
        (object_id, event_id),
    )
    conn.commit()
    return {"success": True, "known_object_id": object_id}


@router.post("/{event_id}/dismiss")
async def dismiss_event(event_id: int):
    conn = get_connection(config.DB_PATH)
    result = conn.execute(
        "UPDATE events SET match_status = 'unknown' WHERE id = ?", (event_id,)
    )
    conn.commit()
    if result.rowcount == 0:
        raise HTTPException(404, "Event not found")
    return {"success": True}

Step 3: Create app/routes/known_objects.py

import base64
from fastapi import APIRouter, HTTPException
from database import get_connection
from config import config
from models import KnownObjectCreate, KnownObjectUpdate

router = APIRouter(prefix="/api/known-objects", tags=["known-objects"])


@router.get("")
async def list_known_objects(type: str | None = None):
    conn = get_connection(config.DB_PATH)
    query = """
        SELECT ko.*, COUNT(e.id) as embedding_count
        FROM known_objects ko
        LEFT JOIN embeddings e ON ko.id = e.known_object_id
    """
    params = []
    if type:
        query += " WHERE ko.type = ?"
        params.append(type)
    query += " GROUP BY ko.id ORDER BY ko.name"
    rows = conn.execute(query, params).fetchall()
    return [dict(row) for row in rows]


@router.get("/{object_id}")
async def get_known_object(object_id: int):
    conn = get_connection(config.DB_PATH)
    obj = conn.execute("SELECT * FROM known_objects WHERE id = ?", (object_id,)).fetchone()
    if not obj:
        raise HTTPException(404, "Known object not found")

    embeddings = conn.execute(
        "SELECT id, created_at FROM embeddings WHERE known_object_id = ?",
        (object_id,),
    ).fetchall()

    result = dict(obj)
    result["embeddings"] = [dict(e) for e in embeddings]
    return result


@router.get("/{object_id}/thumbnails/{embedding_id}")
async def get_thumbnail(object_id: int, embedding_id: int):
    from fastapi.responses import Response

    conn = get_connection(config.DB_PATH)
    row = conn.execute(
        "SELECT thumbnail FROM embeddings WHERE id = ? AND known_object_id = ?",
        (embedding_id, object_id),
    ).fetchone()
    if not row or not row["thumbnail"]:
        raise HTTPException(404, "Thumbnail not found")
    return Response(content=row["thumbnail"], media_type="image/jpeg")


@router.put("/{object_id}")
async def update_known_object(object_id: int, body: KnownObjectUpdate):
    conn = get_connection(config.DB_PATH)
    obj = conn.execute("SELECT * FROM known_objects WHERE id = ?", (object_id,)).fetchone()
    if not obj:
        raise HTTPException(404, "Known object not found")

    updates = {}
    if body.name is not None:
        updates["name"] = body.name
    if body.type is not None:
        updates["type"] = body.type
    if body.action is not None:
        updates["action"] = body.action

    if updates:
        set_clause = ", ".join(f"{k} = ?" for k in updates)
        values = list(updates.values()) + [object_id]
        conn.execute(
            f"UPDATE known_objects SET {set_clause}, updated_at = CURRENT_TIMESTAMP WHERE id = ?",
            values,
        )
        conn.commit()
    return {"success": True}


@router.delete("/{object_id}")
async def delete_known_object(object_id: int):
    conn = get_connection(config.DB_PATH)
    result = conn.execute("DELETE FROM known_objects WHERE id = ?", (object_id,))
    conn.commit()
    if result.rowcount == 0:
        raise HTTPException(404, "Known object not found")
    return {"success": True}

Step 4: Create app/routes/settings.py

import json
from fastapi import APIRouter
from database import get_connection
from config import config
from models import StatsResponse

router = APIRouter(prefix="/api", tags=["settings"])


@router.get("/settings")
async def get_settings():
    conn = get_connection(config.DB_PATH)
    rows = conn.execute("SELECT key, value FROM settings").fetchall()
    result = {row["key"]: row["value"] for row in rows}
    # Add runtime config values
    result["match_high_confidence"] = str(config.MATCH_HIGH_CONFIDENCE)
    result["match_low_confidence"] = str(config.MATCH_LOW_CONFIDENCE)
    result["rate_limit_seconds"] = str(config.RATE_LIMIT_SECONDS)
    result["allowed_labels"] = json.dumps(list(config.ALLOWED_LABELS))
    result["min_score"] = str(config.MIN_SCORE)
    return result


@router.put("/settings")
async def update_settings(body: dict):
    conn = get_connection(config.DB_PATH)
    for key, value in body.items():
        conn.execute(
            "INSERT OR REPLACE INTO settings (key, value) VALUES (?, ?)",
            (key, str(value)),
        )
    conn.commit()
    return {"success": True}


@router.get("/stats")
async def get_stats():
    conn = get_connection(config.DB_PATH)
    total = conn.execute("SELECT COUNT(*) as c FROM events").fetchone()["c"]
    known = conn.execute("SELECT COUNT(*) as c FROM events WHERE match_status='known'").fetchone()["c"]
    unknown = conn.execute("SELECT COUNT(*) as c FROM events WHERE match_status='unknown'").fetchone()["c"]
    unreviewed = conn.execute("SELECT COUNT(*) as c FROM events WHERE match_status='unreviewed'").fetchone()["c"]
    objects = conn.execute("SELECT COUNT(*) as c FROM known_objects").fetchone()["c"]
    today = conn.execute("SELECT COUNT(*) as c FROM events WHERE date(created_at)=date('now')").fetchone()["c"]
    return StatsResponse(
        total_events=total,
        known_matches=known,
        unknown_events=unknown,
        unreviewed=unreviewed,
        known_objects_count=objects,
        events_today=today,
    )

Step 5: Update app/main.py — add router includes after app creation

Add these lines after app = FastAPI(...):

from routes.events import router as events_router
from routes.known_objects import router as known_objects_router
from routes.settings import router as settings_router

app.include_router(events_router)
app.include_router(known_objects_router)
app.include_router(settings_router)

Step 6: Create tests/test_api.py

import os
import sys
import tempfile
import pytest
from fastapi.testclient import TestClient

sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "app"))

from database import init_db


@pytest.fixture
def client(db_path):
    os.environ["DB_PATH"] = db_path
    # Must reimport after setting env
    import importlib
    import config as config_mod
    importlib.reload(config_mod)
    config_mod.config.DB_PATH = db_path

    init_db(db_path)

    from main import app
    return TestClient(app)


def test_health(client):
    resp = client.get("/health")
    assert resp.status_code == 200
    data = resp.json()
    assert data["status"] == "ok"
    assert data["version"] == "2.0.0"


def test_list_events_empty(client):
    resp = client.get("/api/events")
    assert resp.status_code == 200
    assert resp.json() == []


def test_stats_empty(client):
    resp = client.get("/api/stats")
    assert resp.status_code == 200
    data = resp.json()
    assert data["total_events"] == 0


def test_settings_get(client):
    resp = client.get("/api/settings")
    assert resp.status_code == 200
    data = resp.json()
    assert data["vision_backend"] == "haiku"


def test_settings_update(client):
    resp = client.put("/api/settings", json={"vision_backend": "ollama"})
    assert resp.status_code == 200
    resp = client.get("/api/settings")
    assert resp.json()["vision_backend"] == "ollama"


def test_known_objects_crud(client):
    # Create via tag
    from database import get_connection
    conn = get_connection(client.app.state if hasattr(client.app, 'state') else db_path)
    # Use the API - create a known object indirectly
    resp = client.get("/api/known-objects")
    assert resp.status_code == 200
    assert resp.json() == []


def test_event_not_found(client):
    resp = client.get("/api/events/999")
    assert resp.status_code == 404

Step 7: Run tests

pytest tests/ -v

Expected: All tests pass.

Step 8: Commit

git add -A
git commit -m "feat: add REST API endpoints for events, known objects, and settings"
git push origin main

---

Task 3: CLIP Service Sidecar

Goal: Create a standalone FastAPI service that runs on the GPU machine (192.168.86.196), loads the CLIP model, and provides embedding generation endpoints.

Files:

• Create: clip-service/app/main.py
• Create: clip-service/app/requirements.txt
• Create: clip-service/Dockerfile
• Create: clip-service/docker-compose.yml
• Create: clip-service/tests/test_clip.py

Step 1: Create clip-service/app/requirements.txt

fastapi>=0.115.0
uvicorn[standard]>=0.34.0
open-clip-torch>=2.29.0
torch>=2.5.0
Pillow>=10.0.0
numpy>=1.26.0
python-multipart>=0.0.18

Step 2: Create clip-service/app/main.py

import base64
import io
import logging
import sys
import time

import numpy as np
import open_clip
import torch
from fastapi import FastAPI, HTTPException
from PIL import Image
from pydantic import BaseModel

logging.basicConfig(
    level=logging.INFO,
    format='{"time":"%(asctime)s","level":"%(levelname)s","message":"%(message)s"}',
    stream=sys.stdout,
)
logger = logging.getLogger(__name__)

app = FastAPI(title="CLIP Embedding Service", version="1.0.0")

# Global model state
model = None
preprocess = None
tokenizer = None
device = None


class EmbedRequest(BaseModel):
    image: str  # base64-encoded JPEG


class EmbedResponse(BaseModel):
    embedding: list[float]
    inference_ms: float


class SimilarityRequest(BaseModel):
    embedding1: list[float]
    embedding2: list[float]


class SimilarityResponse(BaseModel):
    score: float


class BatchSimilarityRequest(BaseModel):
    query_embedding: list[float]
    candidate_embeddings: list[list[float]]


class BatchSimilarityResponse(BaseModel):
    scores: list[float]
    best_index: int
    best_score: float


@app.on_event("startup")
async def load_model():
    global model, preprocess, tokenizer, device
    device = "cuda" if torch.cuda.is_available() else "cpu"
    logger.info("Loading CLIP model on %s...", device)
    model, _, preprocess = open_clip.create_model_and_transforms(
        "ViT-L-14", pretrained="openai"
    )
    model = model.to(device).eval()
    tokenizer = open_clip.get_tokenizer("ViT-L-14")
    logger.info("CLIP model loaded successfully on %s", device)


@app.get("/health")
async def health():
    return {
        "status": "ok",
        "device": str(device),
        "model": "ViT-L-14",
    }


@app.post("/embed", response_model=EmbedResponse)
async def embed_image(body: EmbedRequest):
    try:
        image_bytes = base64.b64decode(body.image)
        image = Image.open(io.BytesIO(image_bytes)).convert("RGB")
    except Exception as e:
        raise HTTPException(400, f"Invalid image: {e}")

    start = time.time()
    image_tensor = preprocess(image).unsqueeze(0).to(device)
    with torch.no_grad():
        embedding = model.encode_image(image_tensor)
        embedding = embedding / embedding.norm(dim=-1, keepdim=True)
    elapsed_ms = (time.time() - start) * 1000

    embedding_list = embedding.cpu().numpy().flatten().tolist()
    logger.info("Generated embedding in %.1fms (%d dims)", elapsed_ms, len(embedding_list))
    return EmbedResponse(embedding=embedding_list, inference_ms=round(elapsed_ms, 1))


@app.post("/similarity", response_model=SimilarityResponse)
async def compute_similarity(body: SimilarityRequest):
    a = np.array(body.embedding1, dtype=np.float32)
    b = np.array(body.embedding2, dtype=np.float32)
    score = float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))
    return SimilarityResponse(score=score)


@app.post("/batch-similarity", response_model=BatchSimilarityResponse)
async def batch_similarity(body: BatchSimilarityRequest):
    query = np.array(body.query_embedding, dtype=np.float32)
    candidates = np.array(body.candidate_embeddings, dtype=np.float32)
    query_norm = query / np.linalg.norm(query)
    cand_norms = candidates / np.linalg.norm(candidates, axis=1, keepdims=True)
    scores = (cand_norms @ query_norm).tolist()
    best_idx = int(np.argmax(scores))
    return BatchSimilarityResponse(
        scores=scores, best_index=best_idx, best_score=scores[best_idx]
    )

Step 3: Create clip-service/Dockerfile

FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*

COPY app/requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app/ .

EXPOSE 5200

CMD ["python", "-u", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "5200"]

Step 4: Create clip-service/docker-compose.yml

services:
  clip-service:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: clip-service
    restart: unless-stopped
    ports:
      - "5200:5200"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    mem_limit: 4G
    memswap_limit: 4G
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:5200/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"

Step 5: Create clip-service/tests/test_clip.py

import numpy as np


def test_cosine_similarity_identical():
    """Test that identical embeddings have similarity 1.0"""
    a = np.random.randn(768).astype(np.float32)
    a = a / np.linalg.norm(a)
    score = float(np.dot(a, a))
    assert abs(score - 1.0) < 1e-5


def test_cosine_similarity_orthogonal():
    """Test that orthogonal embeddings have similarity ~0"""
    a = np.zeros(768, dtype=np.float32)
    a[0] = 1.0
    b = np.zeros(768, dtype=np.float32)
    b[1] = 1.0
    score = float(np.dot(a, b))
    assert abs(score) < 1e-5


def test_batch_similarity():
    """Test batch similarity returns correct best match"""
    query = np.random.randn(768).astype(np.float32)
    query = query / np.linalg.norm(query)
    # Candidate 0: random (low similarity)
    c0 = np.random.randn(768).astype(np.float32)
    c0 = c0 / np.linalg.norm(c0)
    # Candidate 1: same as query (high similarity)
    c1 = query.copy()
    candidates = np.array([c0, c1])
    query_norm = query / np.linalg.norm(query)
    cand_norms = candidates / np.linalg.norm(candidates, axis=1, keepdims=True)
    scores = (cand_norms @ query_norm).tolist()
    best_idx = int(np.argmax(scores))
    assert best_idx == 1
    assert scores[1] > 0.99

Step 6: Run CLIP tests locally

cd /mnt/c/Users/Jeremy/projects/frigate-notify
pip install numpy
pytest clip-service/tests/ -v

Expected: All 3 tests pass (these are pure numpy tests, no GPU needed).

Step 7: Commit

git add clip-service/
git commit -m "feat: add CLIP embedding service sidecar for GPU machine"
git push origin main

---

Task 4: Recognition Engine

Goal: Add the recognition pipeline that uses CLIP embeddings to match events against known objects, with vision LLM fallback for ambiguous cases.

Files:

• Create: app/recognition.py
• Create: app/ollama_client.py
• Modify: app/vision.py (add Ollama support)
• Create: tests/test_recognition.py

Step 1: Create app/ollama_client.py

import base64
import logging
import aiohttp

logger = logging.getLogger(__name__)


class OllamaClient:
    def __init__(self, base_url: str, model: str):
        self.base_url = base_url.rstrip("/")
        self.model = model

    async def analyze_image(self, image_bytes: bytes, prompt: str) -> str | None:
        """Send image to Ollama vision model for analysis."""
        b64 = base64.b64encode(image_bytes).decode("utf-8")
        payload = {
            "model": self.model,
            "prompt": prompt,
            "images": [b64],
            "stream": False,
            "options": {"num_predict": 200},
        }
        try:
            async with aiohttp.ClientSession() as session:
                async with session.post(
                    f"{self.base_url}/api/generate",
                    json=payload,
                    timeout=aiohttp.ClientTimeout(total=30),
                ) as resp:
                    if resp.status == 200:
                        data = await resp.json()
                        return data.get("response", "").strip()
                    logger.warning("Ollama returned %d", resp.status)
        except Exception:
            logger.exception("Ollama request failed")
        return None

    async def compare_images(
        self, image_bytes: bytes, reference_bytes: bytes, object_name: str
    ) -> bool | None:
        """Ask vision LLM if two images show the same person/vehicle/pet."""
        b64_current = base64.b64encode(image_bytes).decode("utf-8")
        b64_ref = base64.b64encode(reference_bytes).decode("utf-8")
        prompt = (
            f"Look at these two security camera images. The first is a new detection. "
            f"The second is a reference image of '{object_name}'. "
            f"Are they the same person/vehicle/animal? "
            f"Answer ONLY 'yes' or 'no'."
        )
        payload = {
            "model": self.model,
            "prompt": prompt,
            "images": [b64_current, b64_ref],
            "stream": False,
            "options": {"num_predict": 10},
        }
        try:
            async with aiohttp.ClientSession() as session:
                async with session.post(
                    f"{self.base_url}/api/generate",
                    json=payload,
                    timeout=aiohttp.ClientTimeout(total=30),
                ) as resp:
                    if resp.status == 200:
                        data = await resp.json()
                        answer = data.get("response", "").strip().lower()
                        if "yes" in answer:
                            return True
                        if "no" in answer:
                            return False
                    logger.warning("Ollama compare returned %d", resp.status)
        except Exception:
            logger.exception("Ollama compare failed")
        return None

    async def is_available(self) -> bool:
        try:
            async with aiohttp.ClientSession() as session:
                async with session.get(
                    f"{self.base_url}/api/tags",
                    timeout=aiohttp.ClientTimeout(total=3),
                ) as resp:
                    return resp.status == 200
        except Exception:
            return False

Step 2: Create app/recognition.py

import base64
import logging
import struct

import aiohttp
import numpy as np

from config import config
from database import get_connection

logger = logging.getLogger(__name__)


def embedding_to_bytes(embedding: list[float]) -> bytes:
    return struct.pack(f"{len(embedding)}f", *embedding)


def bytes_to_embedding(data: bytes) -> np.ndarray:
    count = len(data) // 4
    return np.array(struct.unpack(f"{count}f", data), dtype=np.float32)


async def get_clip_embedding(image_bytes: bytes) -> list[float] | None:
    """Get CLIP embedding from the clip-service sidecar."""
    b64 = base64.b64encode(image_bytes).decode("utf-8")
    try:
        async with aiohttp.ClientSession() as session:
            async with session.post(
                f"{config.CLIP_SERVICE_URL}/embed",
                json={"image": b64},
                timeout=aiohttp.ClientTimeout(total=10),
            ) as resp:
                if resp.status == 200:
                    data = await resp.json()
                    logger.info("CLIP embedding: %.1fms", data["inference_ms"])
                    return data["embedding"]
                logger.warning("CLIP service returned %d", resp.status)
    except Exception:
        logger.exception("CLIP service unavailable")
    return None


async def find_best_match(
    query_embedding: list[float],
) -> tuple[int | None, float, str | None]:
    """Compare query embedding against all stored embeddings.

    Returns (known_object_id, best_score, object_name) or (None, 0.0, None).
    """
    conn = get_connection(config.DB_PATH)
    rows = conn.execute(
        """SELECT e.known_object_id, e.embedding, ko.name
           FROM embeddings e
           JOIN known_objects ko ON e.known_object_id = ko.id"""
    ).fetchall()

    if not rows:
        return None, 0.0, None

    query = np.array(query_embedding, dtype=np.float32)
    query = query / np.linalg.norm(query)

    best_score = 0.0
    best_object_id = None
    best_name = None

    # Group by object, take max score per object
    object_scores: dict[int, tuple[float, str]] = {}
    for row in rows:
        candidate = bytes_to_embedding(row["embedding"])
        candidate = candidate / np.linalg.norm(candidate)
        score = float(np.dot(query, candidate))
        obj_id = row["known_object_id"]
        if obj_id not in object_scores or score > object_scores[obj_id][0]:
            object_scores[obj_id] = (score, row["name"])

    if object_scores:
        best_object_id = max(object_scores, key=lambda k: object_scores[k][0])
        best_score, best_name = object_scores[best_object_id]

    return best_object_id, best_score, best_name


async def recognize_event(
    image_bytes: bytes, ollama_client=None
) -> tuple[str, int | None, float, str | None]:
    """Run the full recognition pipeline.

    Returns (match_status, matched_object_id, confidence, object_name).
    """
    # Step 1: Get CLIP embedding
    embedding = await get_clip_embedding(image_bytes)
    if embedding is None:
        return "unreviewed", None, 0.0, None

    # Step 2: Find best match
    object_id, score, name = await find_best_match(embedding)

    if object_id is None:
        return "unknown", None, 0.0, None

    # Step 3: Apply confidence thresholds
    if score >= config.MATCH_HIGH_CONFIDENCE:
        logger.info("High confidence match: %s (%.2f)", name, score)
        return "known", object_id, score, name

    if score < config.MATCH_LOW_CONFIDENCE:
        logger.info("Low confidence, treating as unknown (%.2f)", score)
        return "unknown", None, score, None

    # Step 4: Ambiguous — ask vision LLM
    logger.info("Ambiguous match: %s (%.2f), consulting vision LLM", name, score)
    if ollama_client:
        # Get a reference thumbnail for this object
        conn = get_connection(config.DB_PATH)
        ref = conn.execute(
            "SELECT thumbnail FROM embeddings WHERE known_object_id = ? AND thumbnail IS NOT NULL LIMIT 1",
            (object_id,),
        ).fetchone()
        if ref and ref["thumbnail"]:
            result = await ollama_client.compare_images(
                image_bytes, ref["thumbnail"], name
            )
            if result is True:
                logger.info("Vision LLM confirmed: %s", name)
                return "known", object_id, score, name
            elif result is False:
                logger.info("Vision LLM rejected: %s", name)
                return "unknown", None, score, None

    # LLM unavailable or inconclusive — mark ambiguous for human review
    return "ambiguous", object_id, score, name

Step 3: Modify app/vision.py — add Ollama support

import base64
import logging
import anthropic

logger = logging.getLogger(__name__)

PROMPT = (
    "Briefly describe what's happening in this security camera image in 1-2 sentences. "
    "Focus on the people, vehicles, or animals and their activity. Be concise."
)


def analyze_snapshot(client: anthropic.Anthropic, image_bytes: bytes) -> str | None:
    """Send a snapshot to Claude Haiku for description."""
    try:
        b64 = base64.standard_b64encode(image_bytes).decode("utf-8")
        response = client.messages.create(
            model="claude-haiku-4-5-20251001",
            max_tokens=150,
            messages=[
                {
                    "role": "user",
                    "content": [
                        {
                            "type": "image",
                            "source": {
                                "type": "base64",
                                "media_type": "image/jpeg",
                                "data": b64,
                            },
                        },
                        {"type": "text", "text": PROMPT},
                    ],
                }
            ],
        )
        description = response.content[0].text.strip()
        logger.info("Haiku analysis complete: %d chars", len(description))
        return description
    except Exception:
        logger.exception("Claude Vision API failed")
        return None


async def analyze_snapshot_ollama(ollama_client, image_bytes: bytes) -> str | None:
    """Send a snapshot to Ollama vision model for description."""
    return await ollama_client.analyze_image(image_bytes, PROMPT)


async def get_description(
    image_bytes: bytes,
    vision_backend: str,
    claude_client=None,
    ollama_client=None,
) -> str | None:
    """Get scene description using configured backend."""
    if vision_backend == "ollama" and ollama_client:
        result = await analyze_snapshot_ollama(ollama_client, image_bytes)
        if result:
            return result
        # Fall through to haiku if ollama fails
        if claude_client:
            return analyze_snapshot(claude_client, image_bytes)
    elif vision_backend == "haiku" and claude_client:
        return analyze_snapshot(claude_client, image_bytes)
    elif vision_backend == "both" and ollama_client:
        result = await analyze_snapshot_ollama(ollama_client, image_bytes)
        if result:
            return result
        if claude_client:
            return analyze_snapshot(claude_client, image_bytes)
    elif claude_client:
        return analyze_snapshot(claude_client, image_bytes)
    return None

Step 4: Create tests/test_recognition.py

import struct
import numpy as np
from recognition import embedding_to_bytes, bytes_to_embedding


def test_embedding_roundtrip():
    original = np.random.randn(768).astype(np.float32).tolist()
    blob = embedding_to_bytes(original)
    recovered = bytes_to_embedding(blob)
    np.testing.assert_array_almost_equal(recovered, original, decimal=5)


def test_embedding_to_bytes_length():
    embedding = [0.0] * 768
    blob = embedding_to_bytes(embedding)
    assert len(blob) == 768 * 4  # float32 = 4 bytes each


def test_bytes_to_embedding_shape():
    blob = struct.pack("768f", *([0.0] * 768))
    result = bytes_to_embedding(blob)
    assert result.shape == (768,)
    assert result.dtype == np.float32

Step 5: Run tests

pytest tests/ -v

Expected: All tests pass.

Step 6: Commit

git add -A
git commit -m "feat: add recognition engine with CLIP matching and Ollama vision LLM"
git push origin main

---

Task 5: Integrate Recognition into Event Pipeline

Goal: Wire the recognition engine into the MQTT event processing flow so events are automatically matched against known objects before deciding whether to send Matrix alerts.

Files:

• Modify: app/main.py (integrate recognition into process_event)
• Modify: app/docker-compose.yml (add CLIP/Ollama env vars)
• Create: tests/test_event_flow.py

Step 1: Update app/main.py process_event function

Replace the existing process_event function body (after snapshot fetch) with recognition-aware logic. The key change: before sending a Matrix notification, check if recognition is enabled in settings, run the recognition pipeline, and only notify for unknown objects.

Add imports at top of main.py:

from recognition import recognize_event, get_clip_embedding, embedding_to_bytes
from ollama_client import OllamaClient
from vision import get_description

Add ollama_client initialization in startup:

ollama_client = OllamaClient(config.OLLAMA_URL, config.OLLAMA_VISION_MODEL)

Update process_event to accept and use ollama_client, and add recognition logic between snapshot fetch and notification:

    # After fetching snapshot and storing event...

    # Check if recognition is enabled
    recognition_enabled = False
    try:
        row = conn.execute("SELECT value FROM settings WHERE key='recognition_enabled'").fetchone()
        if row:
            recognition_enabled = row["value"].lower() == "true"
    except Exception:
        pass

    # Run recognition if enabled
    match_status = "unreviewed"
    matched_id = None
    match_conf = 0.0
    matched_name = None

    if recognition_enabled and snapshot:
        match_status, matched_id, match_conf, matched_name = await recognize_event(
            snapshot, ollama_client
        )
        # Store embedding
        embedding = await get_clip_embedding(snapshot)
        if embedding:
            embedding_blob = embedding_to_bytes(embedding)
            try:
                conn.execute(
                    "UPDATE events SET embedding=?, match_status=?, matched_object_id=?, match_confidence=? WHERE frigate_event_id=?",
                    (embedding_blob, match_status, matched_id, match_conf, event_id),
                )
                conn.commit()
            except Exception:
                logger.exception("Failed to update event recognition")

        # If known, check action
        if match_status == "known" and matched_id:
            obj = conn.execute("SELECT action FROM known_objects WHERE id=?", (matched_id,)).fetchone()
            if obj and obj["action"] in ("quiet", "suppress"):
                logger.info("Known object '%s' (%s), skipping notification", matched_name, obj["action"])
                events_processed += 1
                last_event_time = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
                return

    # Get AI description using configured backend
    vision_backend = "haiku"
    try:
        row = conn.execute("SELECT value FROM settings WHERE key='vision_backend'").fetchone()
        if row:
            vision_backend = row["value"]
    except Exception:
        pass

    description = await get_description(snapshot, vision_backend, claude, ollama_client) if snapshot else None
    if not description:
        description = f"{label.capitalize()} detected on {camera}."

    # ... rest of notification code unchanged ...

Step 2: Update docker-compose.yml

services:
  frigate-notify:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: frigate-notify
    restart: unless-stopped
    network_mode: host
    environment:
      - TZ=America/Chicago
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - MATRIX_ACCESS_TOKEN=${MATRIX_ACCESS_TOKEN}
      - MATRIX_ROOM_ID=${MATRIX_ROOM_ID}
      - FRIGATE_URL=http://192.168.86.192:5000
      - MQTT_HOST=192.168.86.192
      - MQTT_PORT=1883
      - MATRIX_HOMESERVER=http://192.168.86.167:8008
      - CLIP_SERVICE_URL=http://192.168.86.196:5200
      - OLLAMA_URL=http://192.168.86.196:11434
      - OLLAMA_VISION_MODEL=llava:13b
      - DB_PATH=/data/frigate-notify.db
    volumes:
      - /mnt/docker-data/configs/frigate-notify:/data
    mem_limit: 256M
    memswap_limit: 256M
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:5199/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"

Step 3: Commit

git add -A
git commit -m "feat: integrate recognition engine into MQTT event pipeline"
git push origin main

---

Task 6: Web UI — Dashboard & Event Review

Goal: Create the vanilla HTML/CSS/JS web interface with Dashboard and Review Queue pages.

Files:

• Create: app/static/index.html
• Create: app/static/style.css
• Create: app/static/app.js

Step 1: Create app/static/index.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Frigate Alerts</title>
    <link rel="stylesheet" href="/static/style.css">
</head>
<body>
    <nav id="sidebar">
        <div class="sidebar-header">
            <h2>Frigate Alerts</h2>
        </div>
        <div class="sidebar-section">
            <a href="#" class="sidebar-item active" data-view="dashboard">Dashboard</a>
            <a href="#" class="sidebar-item" data-view="review">Review Queue</a>
            <a href="#" class="sidebar-item" data-view="known">Known Objects</a>
            <a href="#" class="sidebar-item" data-view="settings">Settings</a>
        </div>
        <div class="sidebar-section sidebar-stats">
            <div id="stats-summary"></div>
        </div>
    </nav>
    <main id="content">
        <div id="main-content"></div>
    </main>

    <div id="tag-modal" class="modal hidden">
        <div class="modal-content">
            <h3>Tag as Known Object</h3>
            <div id="tag-modal-snapshot"></div>
            <div class="form-group">
                <label>Existing object:</label>
                <select id="tag-existing-object">
                    <option value="">-- Create new --</option>
                </select>
            </div>
            <div id="tag-new-fields">
                <div class="form-group">
                    <label>Name:</label>
                    <input type="text" id="tag-name" placeholder="e.g., Jeremy, Blue F-150, Max">
                </div>
                <div class="form-group">
                    <label>Type:</label>
                    <select id="tag-type">
                        <option value="person">Person</option>
                        <option value="vehicle">Vehicle</option>
                        <option value="pet">Pet</option>
                    </select>
                </div>
            </div>
            <div class="modal-actions">
                <button id="tag-save" class="btn btn-primary">Save</button>
                <button id="tag-cancel" class="btn">Cancel</button>
            </div>
        </div>
    </div>

    <script src="/static/app.js"></script>
</body>
</html>

Step 2: Create app/static/style.css

* { margin: 0; padding: 0; box-sizing: border-box; }

body {
    font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
    background: #1a1a2e;
    color: #e0e0e0;
    display: flex;
    min-height: 100vh;
}

#sidebar {
    width: 220px;
    background: #16213e;
    padding: 16px 0;
    flex-shrink: 0;
    border-right: 1px solid #0f3460;
}

.sidebar-header { padding: 0 16px 16px; border-bottom: 1px solid #0f3460; }
.sidebar-header h2 { font-size: 18px; color: #e94560; }

.sidebar-section { padding: 8px 0; }

.sidebar-item {
    display: block;
    padding: 10px 16px;
    color: #a0a0b0;
    text-decoration: none;
    font-size: 14px;
    transition: background 0.2s;
}
.sidebar-item:hover { background: #1a1a40; color: #fff; }
.sidebar-item.active { background: #0f3460; color: #e94560; font-weight: 600; }

.sidebar-stats { padding: 16px; font-size: 12px; color: #808090; }

#content { flex: 1; padding: 24px; overflow-y: auto; }

.page-title { font-size: 24px; margin-bottom: 16px; }

.filters { display: flex; gap: 12px; margin-bottom: 16px; flex-wrap: wrap; }
.filters select, .filters input {
    background: #16213e; color: #e0e0e0; border: 1px solid #0f3460;
    padding: 6px 10px; border-radius: 4px; font-size: 13px;
}

.event-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(300px, 1fr)); gap: 16px; }

.event-card {
    background: #16213e;
    border: 1px solid #0f3460;
    border-radius: 8px;
    overflow: hidden;
    transition: border-color 0.2s;
}
.event-card:hover { border-color: #e94560; }

.event-card img {
    width: 100%;
    height: 180px;
    object-fit: cover;
    background: #0a0a1a;
}

.event-card .event-info { padding: 12px; }
.event-card .event-header { display: flex; justify-content: space-between; margin-bottom: 6px; }
.event-card .event-camera { font-weight: 600; font-size: 14px; }
.event-card .event-label {
    font-size: 12px; padding: 2px 8px; border-radius: 12px;
    background: #0f3460; color: #e0e0e0;
}
.event-card .event-description { font-size: 13px; color: #a0a0b0; margin: 6px 0; }
.event-card .event-meta { font-size: 11px; color: #606070; }

.event-card .event-actions { display: flex; gap: 8px; margin-top: 8px; }

.status-known { border-left: 3px solid #4caf50; }
.status-unknown { border-left: 3px solid #e94560; }
.status-ambiguous { border-left: 3px solid #ff9800; }
.status-unreviewed { border-left: 3px solid #2196f3; }

.btn {
    padding: 6px 14px; border: none; border-radius: 4px; cursor: pointer;
    font-size: 13px; transition: background 0.2s;
}
.btn-primary { background: #e94560; color: #fff; }
.btn-primary:hover { background: #c73450; }
.btn-success { background: #4caf50; color: #fff; }
.btn-success:hover { background: #3d9140; }
.btn-danger { background: #f44336; color: #fff; }
.btn { background: #0f3460; color: #e0e0e0; }
.btn:hover { background: #1a4a7a; }

.known-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(250px, 1fr)); gap: 16px; }

.known-card {
    background: #16213e; border: 1px solid #0f3460; border-radius: 8px; padding: 16px;
}
.known-card h4 { margin-bottom: 8px; }
.known-card .known-type { font-size: 12px; color: #a0a0b0; margin-bottom: 8px; }
.known-card .known-thumbs { display: flex; gap: 4px; flex-wrap: wrap; margin: 8px 0; }
.known-card .known-thumbs img { width: 50px; height: 50px; object-fit: cover; border-radius: 4px; }

.settings-form { max-width: 500px; }
.form-group { margin-bottom: 16px; }
.form-group label { display: block; margin-bottom: 4px; font-size: 13px; color: #a0a0b0; }
.form-group select, .form-group input {
    width: 100%; background: #16213e; color: #e0e0e0; border: 1px solid #0f3460;
    padding: 8px 10px; border-radius: 4px; font-size: 14px;
}

.modal { position: fixed; inset: 0; background: rgba(0,0,0,0.7); display: flex; align-items: center; justify-content: center; z-index: 100; }
.modal.hidden { display: none; }
.modal-content { background: #16213e; border-radius: 8px; padding: 24px; min-width: 400px; max-width: 500px; }
.modal-content h3 { margin-bottom: 16px; }
.modal-content img { width: 100%; max-height: 200px; object-fit: cover; border-radius: 4px; margin-bottom: 12px; }
.modal-actions { display: flex; gap: 8px; justify-content: flex-end; margin-top: 16px; }

.empty-state { text-align: center; padding: 60px 20px; color: #606070; }
.empty-state h3 { margin-bottom: 8px; color: #a0a0b0; }

Step 3: Create app/static/app.js

This is the main UI logic. It handles navigation, data fetching, rendering, and the tag modal. The file will be ~400 lines. Key sections:

let currentView = "dashboard";
let eventsData = [];
let knownObjects = [];
let stats = {};

// --- Navigation ---
document.querySelectorAll(".sidebar-item").forEach(item => {
    item.addEventListener("click", e => {
        e.preventDefault();
        currentView = item.dataset.view;
        document.querySelectorAll(".sidebar-item").forEach(i => i.classList.remove("active"));
        item.classList.add("active");
        renderMain();
    });
});

// --- Data fetching ---
async function fetchEvents(params = {}) {
    const query = new URLSearchParams(params).toString();
    const resp = await fetch(`/api/events?${query}`);
    return resp.json();
}

async function fetchEvent(id) {
    const resp = await fetch(`/api/events/${id}`);
    return resp.json();
}

async function fetchKnownObjects() {
    const resp = await fetch("/api/known-objects");
    return resp.json();
}

async function fetchStats() {
    const resp = await fetch("/api/stats");
    return resp.json();
}

async function fetchSettings() {
    const resp = await fetch("/api/settings");
    return resp.json();
}

// --- Rendering ---
async function renderMain() {
    const container = document.getElementById("main-content");
    stats = await fetchStats();
    renderStats();

    switch (currentView) {
        case "dashboard": await renderDashboard(container); break;
        case "review": await renderReview(container); break;
        case "known": await renderKnown(container); break;
        case "settings": await renderSettings(container); break;
    }
}

function renderStats() {
    document.getElementById("stats-summary").innerHTML = `
        <div>Today: ${stats.events_today} events</div>
        <div>Known: ${stats.known_matches}</div>
        <div>Unknown: ${stats.unknown_events}</div>
        <div>Review: ${stats.unreviewed}</div>
    `;
}

async function renderDashboard(container) {
    eventsData = await fetchEvents({ limit: 50 });
    container.innerHTML = `
        <h1 class="page-title">Dashboard</h1>
        <div class="filters">
            <select id="filter-camera" onchange="applyFilters()">
                <option value="">All Cameras</option>
                ${[...new Set(eventsData.map(e => e.camera))].map(c =>
                    `<option value="${c}">${c}</option>`
                ).join("")}
            </select>
            <select id="filter-status" onchange="applyFilters()">
                <option value="">All Statuses</option>
                <option value="known">Known</option>
                <option value="unknown">Unknown</option>
                <option value="ambiguous">Ambiguous</option>
                <option value="unreviewed">Unreviewed</option>
            </select>
        </div>
        <div class="event-grid" id="event-grid"></div>
    `;
    renderEventGrid(eventsData);
}

function renderEventGrid(events) {
    const grid = document.getElementById("event-grid");
    if (!events.length) {
        grid.innerHTML = '<div class="empty-state"><h3>No events</h3><p>Events will appear here when Frigate detects objects.</p></div>';
        return;
    }
    grid.innerHTML = events.map(e => `
        <div class="event-card status-${e.match_status}">
            ${e.has_snapshot
                ? `<img src="/api/events/${e.id}/snapshot" alt="${e.camera}">`
                : '<div style="height:180px;background:#0a0a1a;display:flex;align-items:center;justify-content:center;color:#606070">No snapshot</div>'}
            <div class="event-info">
                <div class="event-header">
                    <span class="event-camera">${e.camera}</span>
                    <span class="event-label">${e.label} ${Math.round(e.score * 100)}%</span>
                </div>
                <div class="event-description">${e.ai_description || ""}</div>
                ${e.matched_object_name ? `<div class="event-meta">Matched: ${e.matched_object_name} (${Math.round((e.match_confidence || 0) * 100)}%)</div>` : ""}
                <div class="event-meta">${new Date(e.created_at).toLocaleString()}</div>
                <div class="event-actions">
                    <button class="btn btn-success" onclick="openTagModal(${e.id})">Tag</button>
                    <button class="btn" onclick="dismissEvent(${e.id})">Dismiss</button>
                </div>
            </div>
        </div>
    `).join("");
}

async function applyFilters() {
    const camera = document.getElementById("filter-camera").value;
    const status = document.getElementById("filter-status").value;
    const params = { limit: 50 };
    if (camera) params.camera = camera;
    if (status) params.status = status;
    const filtered = await fetchEvents(params);
    renderEventGrid(filtered);
}

async function renderReview(container) {
    const events = await fetchEvents({ status: "unreviewed", limit: 50 });
    container.innerHTML = `
        <h1 class="page-title">Review Queue (${events.length})</h1>
        <div class="event-grid" id="event-grid"></div>
    `;
    renderEventGrid(events);
}

async function renderKnown(container) {
    knownObjects = await fetchKnownObjects();
    container.innerHTML = `
        <h1 class="page-title">Known Objects (${knownObjects.length})</h1>
        <div class="known-grid">
            ${knownObjects.length === 0
                ? '<div class="empty-state"><h3>No known objects</h3><p>Tag events from the Dashboard or Review Queue to add objects here.</p></div>'
                : knownObjects.map(obj => `
                    <div class="known-card">
                        <h4>${obj.name}</h4>
                        <div class="known-type">${obj.type} &middot; ${obj.embedding_count} samples</div>
                        <div class="event-actions">
                            <button class="btn btn-danger" onclick="deleteKnownObject(${obj.id})">Delete</button>
                        </div>
                    </div>
                `).join("")}
        </div>
    `;
}

async function renderSettings(container) {
    const settings = await fetchSettings();
    container.innerHTML = `
        <h1 class="page-title">Settings</h1>
        <div class="settings-form">
            <div class="form-group">
                <label>Vision Backend (scene descriptions)</label>
                <select id="setting-vision" onchange="saveSetting('vision_backend', this.value)">
                    <option value="haiku" ${settings.vision_backend === "haiku" ? "selected" : ""}>Claude Haiku (cloud)</option>
                    <option value="ollama" ${settings.vision_backend === "ollama" ? "selected" : ""}>Ollama (local GPU)</option>
                    <option value="both" ${settings.vision_backend === "both" ? "selected" : ""}>Ollama primary, Haiku fallback</option>
                </select>
            </div>
            <div class="form-group">
                <label>Recognition (known object matching)</label>
                <select id="setting-recognition" onchange="saveSetting('recognition_enabled', this.value)">
                    <option value="true" ${settings.recognition_enabled === "true" ? "selected" : ""}>Enabled</option>
                    <option value="false" ${settings.recognition_enabled === "false" ? "selected" : ""}>Disabled</option>
                </select>
            </div>
            <div class="form-group">
                <label>Event Retention (days)</label>
                <input type="number" value="${settings.event_retention_days || 90}"
                    onchange="saveSetting('event_retention_days', this.value)">
            </div>
        </div>
    `;
}

// --- Actions ---
async function dismissEvent(id) {
    await fetch(`/api/events/${id}/dismiss`, { method: "POST" });
    renderMain();
}

async function deleteKnownObject(id) {
    if (!confirm("Delete this known object and all its reference data?")) return;
    await fetch(`/api/known-objects/${id}`, { method: "DELETE" });
    renderMain();
}

async function saveSetting(key, value) {
    await fetch("/api/settings", {
        method: "PUT",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ [key]: value }),
    });
}

// --- Tag Modal ---
let tagEventId = null;

async function openTagModal(eventId) {
    tagEventId = eventId;
    const modal = document.getElementById("tag-modal");
    const snapshotDiv = document.getElementById("tag-modal-snapshot");
    snapshotDiv.innerHTML = `<img src="/api/events/${eventId}/snapshot">`;

    // Populate existing objects dropdown
    const objects = await fetchKnownObjects();
    const select = document.getElementById("tag-existing-object");
    select.innerHTML = '<option value="">-- Create new --</option>' +
        objects.map(o => `<option value="${o.id}">${o.name} (${o.type})</option>`).join("");

    select.onchange = () => {
        document.getElementById("tag-new-fields").style.display = select.value ? "none" : "block";
    };
    document.getElementById("tag-new-fields").style.display = "block";
    modal.classList.remove("hidden");
}

document.getElementById("tag-cancel").addEventListener("click", () => {
    document.getElementById("tag-modal").classList.add("hidden");
});

document.getElementById("tag-save").addEventListener("click", async () => {
    const existingId = document.getElementById("tag-existing-object").value;
    let body;
    if (existingId) {
        body = { known_object_id: parseInt(existingId) };
    } else {
        const name = document.getElementById("tag-name").value.trim();
        const type = document.getElementById("tag-type").value;
        if (!name) { alert("Please enter a name"); return; }
        body = { name, type };
    }
    await fetch(`/api/events/${tagEventId}/tag`, {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify(body),
    });
    document.getElementById("tag-modal").classList.add("hidden");
    renderMain();
});

// --- Auto-refresh ---
setInterval(() => { if (currentView === "dashboard") renderMain(); }, 30000);

// --- Init ---
renderMain();

Step 4: Commit

git add app/static/
git commit -m "feat: add web UI with dashboard, review queue, known objects, and settings pages"
git push origin main

---

Task 7: Event Retention & Cleanup

Goal: Add automatic cleanup of old events based on the retention setting.

Files:

• Create: app/cleanup.py
• Modify: app/main.py (schedule cleanup on startup)
• Create: tests/test_cleanup.py

Step 1: Create app/cleanup.py

import asyncio
import logging
from database import get_connection
from config import config

logger = logging.getLogger(__name__)


async def cleanup_old_events():
    """Delete events older than the configured retention period."""
    while True:
        try:
            conn = get_connection(config.DB_PATH)
            row = conn.execute(
                "SELECT value FROM settings WHERE key='event_retention_days'"
            ).fetchone()
            days = int(row["value"]) if row else 90

            result = conn.execute(
                "DELETE FROM events WHERE created_at < datetime('now', ? || ' days')",
                (f"-{days}",),
            )
            conn.commit()
            if result.rowcount > 0:
                logger.info("Cleaned up %d events older than %d days", result.rowcount, days)
        except Exception:
            logger.exception("Cleanup failed")

        await asyncio.sleep(3600)  # Run every hour

Step 2: Add to app/main.py startup

from cleanup import cleanup_old_events

@app.on_event("startup")
async def startup():
    init_db(config.DB_PATH)
    start_mqtt()
    asyncio.create_task(cleanup_old_events())
    logger.info("Frigate-notify v2 started")

Step 3: Create tests/test_cleanup.py

import sqlite3
from database import init_db


def test_cleanup_deletes_old_events(db_path):
    init_db(db_path)
    conn = sqlite3.connect(db_path)
    conn.row_factory = sqlite3.Row
    # Insert an old event
    conn.execute(
        """INSERT INTO events (frigate_event_id, camera, label, score, match_status, created_at)
           VALUES ('old-1', 'cam', 'person', 0.9, 'unknown', datetime('now', '-100 days'))"""
    )
    # Insert a recent event
    conn.execute(
        """INSERT INTO events (frigate_event_id, camera, label, score, match_status)
           VALUES ('new-1', 'cam', 'person', 0.9, 'unknown')"""
    )
    conn.commit()

    result = conn.execute(
        "DELETE FROM events WHERE created_at < datetime('now', '-90 days')"
    )
    conn.commit()
    assert result.rowcount == 1

    remaining = conn.execute("SELECT COUNT(*) as c FROM events").fetchone()["c"]
    assert remaining == 1
    conn.close()

Step 4: Run tests and commit

pytest tests/ -v
git add -A
git commit -m "feat: add automatic event cleanup based on retention setting"
git push origin main

---

Task 8: Farm Manager Sidebar Link

Goal: Add a "Frigate Alerts" link to the Farm Manager "Server Links" page.

Files:

• Modify: /mnt/c/Users/Jeremy/projects/farm-manager/server/static/app.js (add link to serverLinks array)

Step 1: Add link entry

In app.js around line 734, the serverLinks array has categories. Add to the appropriate node category (hf-pdocker-01 section):

{ name: "Frigate Alerts", url: "http://192.168.86.192:5199", icon: "\uD83D\uDD14" },

This adds a bell icon link to the frigate-notify v2 web UI.

Step 2: Commit in the farm-manager repo

cd /mnt/c/Users/Jeremy/projects/farm-manager
git add server/static/app.js
git commit -m "feat: add Frigate Alerts link to server links page"
git push origin main

---

Task 9: Pull Ollama Vision Model

Goal: Pull llava:13b on the GPU machine so vision LLM is ready.

Step 1: Pull the model

curl -X POST http://192.168.86.196:11434/api/pull -d '{"name":"llava:13b"}'

This will take several minutes to download (~8GB). Verify with:

curl http://192.168.86.196:11434/api/tags | python3 -m json.tool

Expected: llava:13b appears in the model list.

---

Task 10: Deploy clip-service to GPU Machine

Goal: Build and deploy the CLIP service container on 192.168.86.196.

Step 1: Copy clip-service files to GPU machine

scp -r /mnt/c/Users/Jeremy/projects/frigate-notify/clip-service/ glyptic@192.168.86.196:/tmp/clip-service/

Step 2: Build and start on GPU machine

ssh glyptic@192.168.86.196
cd /tmp/clip-service
docker compose up -d --build

Step 3: Verify

curl http://192.168.86.196:5200/health

Expected: {"status":"ok","device":"cuda","model":"ViT-L-14"}

Step 4: Test embedding generation

# Create a small test image (1x1 pixel JPEG)
python3 -c "
import base64, json
# Use a real test snapshot if available, or a minimal JPEG
with open('/path/to/test.jpg', 'rb') as f:
    b64 = base64.b64encode(f.read()).decode()
import urllib.request
req = urllib.request.Request(
    'http://192.168.86.196:5200/embed',
    data=json.dumps({'image': b64}).encode(),
    headers={'Content-Type': 'application/json'}
)
resp = urllib.request.urlopen(req)
data = json.loads(resp.read())
print(f'Embedding dims: {len(data[\"embedding\"])}, time: {data[\"inference_ms\"]}ms')
"

Expected: 768-dimensional embedding, <100ms inference time.

---

Task 11: Deploy frigate-notify v2

Goal: Build and deploy the updated frigate-notify on hf-pdocker-01 (when it's back online).

NOTE: This task is blocked until hf-pdocker-01 is restored. The compose file and code are ready on the NAS.

Step 1: Create data directory on NAS

ssh glyptic@192.168.86.100
mkdir -p /mnt/docker-data/configs/frigate-notify

Step 2: Copy updated code to NAS

# From local machine, sync to NAS via any node
scp -r /mnt/c/Users/Jeremy/projects/frigate-notify/{app,Dockerfile,docker-compose.yml} \
    glyptic@192.168.86.100:/mnt/docker-data/compose/hf-pdocker-01/frigate-notify/

Step 3: Deploy on hf-pdocker-01 (when available)

ssh glyptic@192.168.86.192
cd /mnt/docker-data/compose/hf-pdocker-01/frigate-notify
docker compose up -d --build

Step 4: Verify

curl http://192.168.86.192:5199/health
# Open http://192.168.86.192:5199 in browser — should show Dashboard

---

Summary

Task	Component	Dependencies
1	Project scaffold, FastAPI, SQLite	None
2	REST API endpoints	Task 1
3	CLIP service sidecar	None (independent)
4	Recognition engine	Task 1
5	Pipeline integration	Tasks 2, 4
6	Web UI	Task 2
7	Event cleanup	Task 1
8	Farm Manager link	None (independent)
9	Pull Ollama model	None (independent)
10	Deploy clip-service	Task 3
11	Deploy frigate-notify v2	Tasks 1-7, hf-pdocker-01 online

Independent tasks (can run in parallel): 1, 3, 8, 9 Sequential chain: 1 → 2 → 4 → 5 → 6 → 7 → 11