# Frigate-Notify v2 Implementation Plan > **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. **Goal:** Evolve frigate-notify into an intelligent notification system with known-object recognition (CLIP + Vision LLM), a web UI for managing identities and reviewing events, and configurable vision backends (local Ollama or Claude Haiku). **Architecture:** A standalone Python/FastAPI web app (:5199) that listens for Frigate MQTT events, matches snapshots against a known-objects database using CLIP embeddings (via a GPU sidecar on 192.168.86.196), falls back to vision LLM for ambiguous matches, and only sends Matrix alerts for unknown objects. A vanilla HTML/CSS/JS web UI provides event review, object tagging, and settings management. **Tech Stack:** Python 3.12, FastAPI, SQLite, paho-mqtt, aiohttp, open_clip (sidecar), Ollama, Claude Haiku API, vanilla HTML/CSS/JS **Design Doc:** `docs/plans/2026-03-07-frigate-notify-v2-design.md` --- ## Context for the Implementer ### Existing Code The v1 app lives at `/mnt/docker-data/compose/hf-pdocker-01/frigate-notify/` on the NAS and in the Gitea repo `jeremy/frigate-notify`. The source is also in the local repo at `/mnt/c/Users/Jeremy/projects/docker/compose/hf-pdocker-01/frigate-notify/`. ``` frigate-notify/ docker-compose.yml # Compose definition Dockerfile # python:3.12-slim based .env # ANTHROPIC_API_KEY, MATRIX_ACCESS_TOKEN, MATRIX_ROOM_ID app/ main.py # MQTT listener, event processor, health endpoint (~200 lines) vision.py # Claude Haiku vision wrapper (~30 lines) matrix_client.py # Matrix API client (~80 lines) requirements.txt # paho-mqtt, anthropic, aiohttp ``` ### Key Services | Service | Host | Port | Purpose | |---------|------|------|---------| | Frigate NVR | 192.168.86.192 (hf-pdocker-01) | 5000 | Camera NVR, snapshots API | | Mosquitto | 192.168.86.192 | 1883 | MQTT broker | | Matrix/Synapse | 192.168.86.167 (bart, failover) | 8008 | Chat server | | Ollama | 192.168.86.196 | 11434 | Local LLM inference (RTX 3090) | | Farm Manager | varies | 8888 | Dashboard (sidebar link target) | ### Git Repo - **Gitea**: `http://192.168.86.100:3003/jeremy/frigate-notify` - **Clone URL**: `http://jeremy:806153c545dc3b227ecdb86b134a2e50210a8e89@192.168.86.100:3003/jeremy/frigate-notify.git` - **Local working copy**: `/mnt/c/Users/Jeremy/projects/frigate-notify/` (create if doesn't exist) ### SSH Access (for deployment) - Username: `glyptic`, Password: `&Rip8a115` - Use: `sshpass -p '&Rip8a115' ssh -o StrictHostKeyChecking=no glyptic@` ### Note on hf-pdocker-01 hf-pdocker-01 is currently DOWN (hard drive failure). Frigate-notify can't run until it's restored (it needs Frigate + MQTT on that node). Development and testing should be done locally and via the Gitea repo. The clip-service sidecar on 192.168.86.196 CAN be deployed and tested independently. --- ## Task 1: Project Scaffolding & FastAPI Migration **Goal:** Restructure the project for v2 — migrate from raw HTTP to FastAPI, add proper project layout with tests, set up SQLite database. **Files:** - Modify: `app/main.py` (replace HTTPServer with FastAPI) - Create: `app/database.py` (SQLite schema and connection) - Create: `app/models.py` (Pydantic models) - Create: `app/config.py` (centralized configuration) - Modify: `app/requirements.txt` (add fastapi, uvicorn, python-multipart) - Create: `tests/conftest.py` - Create: `tests/test_database.py` - Modify: `Dockerfile` (change CMD to uvicorn) **Step 1: Clone the repo locally and set up project structure** ```bash cd /mnt/c/Users/Jeremy/projects git clone http://jeremy:806153c545dc3b227ecdb86b134a2e50210a8e89@192.168.86.100:3003/jeremy/frigate-notify.git cd frigate-notify ``` **Step 2: Create `app/config.py`** ```python import os class Config: # MQTT MQTT_HOST: str = os.environ.get("MQTT_HOST", "192.168.86.192") MQTT_PORT: int = int(os.environ.get("MQTT_PORT", "1883")) # Frigate FRIGATE_URL: str = os.environ.get("FRIGATE_URL", "http://192.168.86.192:5000") # Matrix MATRIX_HOMESERVER: str = os.environ.get("MATRIX_HOMESERVER", "http://192.168.86.192:8008") MATRIX_ACCESS_TOKEN: str = os.environ.get("MATRIX_ACCESS_TOKEN", "") MATRIX_ROOM_ID: str = os.environ.get("MATRIX_ROOM_ID", "") # Vision backends CLIP_SERVICE_URL: str = os.environ.get("CLIP_SERVICE_URL", "http://192.168.86.196:5200") OLLAMA_URL: str = os.environ.get("OLLAMA_URL", "http://192.168.86.196:11434") OLLAMA_VISION_MODEL: str = os.environ.get("OLLAMA_VISION_MODEL", "llava:13b") # Recognition thresholds MATCH_HIGH_CONFIDENCE: float = float(os.environ.get("MATCH_HIGH_CONFIDENCE", "0.85")) MATCH_LOW_CONFIDENCE: float = float(os.environ.get("MATCH_LOW_CONFIDENCE", "0.60")) # Filtering ALLOWED_LABELS: set = {"person", "car", "dog", "cat", "bird"} MIN_SCORE: float = 0.7 RATE_LIMIT_SECONDS: int = 60 DEDUP_WINDOW_SECONDS: int = 30 # Server HOST: str = "0.0.0.0" PORT: int = 5199 # Database DB_PATH: str = os.environ.get("DB_PATH", "/data/frigate-notify.db") config = Config() ``` **Step 3: Create `app/database.py`** ```python import sqlite3 import threading from contextlib import contextmanager _local = threading.local() def get_connection(db_path: str) -> sqlite3.Connection: if not hasattr(_local, "connection") or _local.connection is None: _local.connection = sqlite3.connect(db_path, check_same_thread=False) _local.connection.row_factory = sqlite3.Row _local.connection.execute("PRAGMA journal_mode=WAL") _local.connection.execute("PRAGMA foreign_keys=ON") return _local.connection def init_db(db_path: str) -> None: conn = sqlite3.connect(db_path) conn.executescript(""" CREATE TABLE IF NOT EXISTS known_objects ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL, type TEXT NOT NULL CHECK(type IN ('person', 'vehicle', 'pet')), action TEXT NOT NULL DEFAULT 'quiet' CHECK(action IN ('quiet', 'suppress', 'alert')), created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE IF NOT EXISTS embeddings ( id INTEGER PRIMARY KEY AUTOINCREMENT, known_object_id INTEGER NOT NULL REFERENCES known_objects(id) ON DELETE CASCADE, embedding BLOB NOT NULL, snapshot_url TEXT, thumbnail BLOB, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE IF NOT EXISTS events ( id INTEGER PRIMARY KEY AUTOINCREMENT, frigate_event_id TEXT UNIQUE NOT NULL, camera TEXT NOT NULL, label TEXT NOT NULL, score REAL NOT NULL, snapshot BLOB, embedding BLOB, match_status TEXT NOT NULL DEFAULT 'unreviewed' CHECK(match_status IN ('known', 'unknown', 'ambiguous', 'unreviewed')), matched_object_id INTEGER REFERENCES known_objects(id) ON DELETE SET NULL, match_confidence REAL, ai_description TEXT, notified INTEGER NOT NULL DEFAULT 0, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE IF NOT EXISTS settings ( key TEXT PRIMARY KEY, value TEXT NOT NULL ); CREATE INDEX IF NOT EXISTS idx_events_created ON events(created_at); CREATE INDEX IF NOT EXISTS idx_events_status ON events(match_status); CREATE INDEX IF NOT EXISTS idx_events_camera ON events(camera); CREATE INDEX IF NOT EXISTS idx_embeddings_object ON embeddings(known_object_id); """) # Insert default settings if not present defaults = { "vision_backend": "haiku", "recognition_enabled": "false", "event_retention_days": "90", } for key, value in defaults.items(): conn.execute( "INSERT OR IGNORE INTO settings (key, value) VALUES (?, ?)", (key, value), ) conn.commit() conn.close() ``` **Step 4: Create `app/models.py`** ```python from pydantic import BaseModel from typing import Optional from datetime import datetime class KnownObjectCreate(BaseModel): name: str type: str # person, vehicle, pet action: str = "quiet" # quiet, suppress, alert class KnownObjectUpdate(BaseModel): name: Optional[str] = None type: Optional[str] = None action: Optional[str] = None class KnownObjectResponse(BaseModel): id: int name: str type: str action: str created_at: str updated_at: str embedding_count: int = 0 class TagEventRequest(BaseModel): known_object_id: Optional[int] = None name: Optional[str] = None type: Optional[str] = None class EventResponse(BaseModel): id: int frigate_event_id: str camera: str label: str score: float match_status: str matched_object_id: Optional[int] = None matched_object_name: Optional[str] = None match_confidence: Optional[float] = None ai_description: Optional[str] = None notified: bool created_at: str class SettingsResponse(BaseModel): vision_backend: str = "haiku" recognition_enabled: bool = False event_retention_days: int = 90 match_high_confidence: float = 0.85 match_low_confidence: float = 0.60 rate_limit_seconds: int = 60 allowed_labels: list[str] = ["person", "car", "dog", "cat", "bird"] min_score: float = 0.7 class StatsResponse(BaseModel): total_events: int known_matches: int unknown_events: int unreviewed: int known_objects_count: int events_today: int ``` **Step 5: Update `app/requirements.txt`** ``` paho-mqtt==2.1.0 anthropic>=0.40.0 aiohttp>=3.9.0 fastapi>=0.115.0 uvicorn[standard]>=0.34.0 python-multipart>=0.0.18 numpy>=1.26.0 ``` **Step 6: Rewrite `app/main.py` for FastAPI** This is the big refactor. The new main.py sets up FastAPI, mounts the MQTT listener as a background task, and serves both the API and static files. ```python import asyncio import json import logging import os import signal import sys import time from collections import defaultdict from threading import Thread import aiohttp import anthropic import paho.mqtt.client as mqtt from fastapi import FastAPI from fastapi.staticfiles import StaticFiles from fastapi.responses import FileResponse from config import config from database import init_db, get_connection from vision import analyze_snapshot from matrix_client import MatrixClient # --- Logging --- logging.basicConfig( level=logging.INFO, format='{"time":"%(asctime)s","level":"%(levelname)s","module":"%(module)s","message":"%(message)s"}', stream=sys.stdout, ) logger = logging.getLogger(__name__) # --- State --- recent_events: dict[str, float] = {} last_notify_per_camera: defaultdict[str, float] = defaultdict(float) events_processed = 0 last_event_time: str | None = None mqtt_connected = False # --- App --- app = FastAPI(title="Frigate Notify", version="2.0.0") # Mount static files for web UI static_dir = os.path.join(os.path.dirname(__file__), "static") if os.path.isdir(static_dir): app.mount("/static", StaticFiles(directory=static_dir), name="static") @app.on_event("startup") async def startup(): init_db(config.DB_PATH) start_mqtt() logger.info("Frigate-notify v2 started") @app.get("/health") async def health(): return { "status": "ok", "version": "2.0.0", "mqtt_connected": mqtt_connected, "events_processed": events_processed, "last_event": last_event_time, } @app.get("/") async def index(): return FileResponse(os.path.join(static_dir, "index.html")) # --- Snapshot fetching --- async def fetch_snapshot(event_id: str) -> bytes | None: url = f"{config.FRIGATE_URL}/api/events/{event_id}/snapshot.jpg" try: async with aiohttp.ClientSession() as session: async with session.get(url) as resp: if resp.status == 200: data = await resp.read() logger.info("Fetched snapshot for %s: %d bytes", event_id, len(data)) return data logger.warning("Snapshot fetch failed for %s: HTTP %d", event_id, resp.status) except Exception: logger.exception("Snapshot fetch error for %s", event_id) return None # --- Event processing --- async def process_event(event_data: dict, claude: anthropic.Anthropic, matrix: MatrixClient): global events_processed, last_event_time event_type = event_data.get("type") if event_type != "end": return after = event_data.get("after", {}) event_id = after.get("id", "") camera = after.get("camera", "unknown") label = after.get("label", "") score = after.get("top_score", after.get("score", 0)) has_snapshot = after.get("has_snapshot", False) if label not in config.ALLOWED_LABELS: return if score < config.MIN_SCORE: return if not has_snapshot: return now = time.time() if event_id in recent_events and now - recent_events[event_id] < config.DEDUP_WINDOW_SECONDS: return recent_events[event_id] = now if now - last_notify_per_camera[camera] < config.RATE_LIMIT_SECONDS: logger.info("Rate limited: %s on %s", label, camera) return last_notify_per_camera[camera] = now cutoff = now - config.DEDUP_WINDOW_SECONDS * 2 for eid in [k for k, v in recent_events.items() if v < cutoff]: del recent_events[eid] logger.info("Processing event: %s on %s (%s, score=%.2f)", event_id, camera, label, score) snapshot = await fetch_snapshot(event_id) # Store event in database snapshot_thumb = None if snapshot: # Store a smaller thumbnail (first 100KB max) snapshot_thumb = snapshot[:102400] if len(snapshot) > 102400 else snapshot conn = get_connection(config.DB_PATH) try: conn.execute( """INSERT OR IGNORE INTO events (frigate_event_id, camera, label, score, snapshot, match_status) VALUES (?, ?, ?, ?, ?, 'unreviewed')""", (event_id, camera, label, score, snapshot_thumb), ) conn.commit() except Exception: logger.exception("Failed to store event %s", event_id) # AI description description = None if snapshot: description = analyze_snapshot(claude, snapshot) if not description: description = f"{label.capitalize()} detected on {camera}." # Update event with description try: conn.execute( "UPDATE events SET ai_description = ? WHERE frigate_event_id = ?", (description, event_id), ) conn.commit() except Exception: logger.exception("Failed to update event description") # Format and send Matrix notification score_pct = int(score * 100) ts = time.strftime("%I:%M:%S %p", time.localtime()) date_str = time.strftime("%b %d, %Y", time.localtime()) frigate_link = f"{config.FRIGATE_URL}/review?id={event_id}" plain = ( f"{camera} - {label.capitalize()} - {score_pct}%\n" f"{description}\n\n" f"View in Frigate: {frigate_link}\n" f"{ts} - {date_str}" ) html = ( f"{camera} · {label.capitalize()} · {score_pct}%
" f"{description}

" f'View in Frigate
' f"{ts} · {date_str}" ) if snapshot: filename = f"{camera}_{event_id}.jpg" mxc_uri = await matrix.upload_image(snapshot, filename) if mxc_uri: await matrix.send_image(mxc_uri, filename, len(snapshot)) await matrix.send_text(plain, html) # Mark as notified try: conn.execute( "UPDATE events SET notified = 1 WHERE frigate_event_id = ?", (event_id,), ) conn.commit() except Exception: pass events_processed += 1 last_event_time = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) logger.info("Notification sent for %s on %s", label, camera) # --- MQTT --- def start_mqtt(): global mqtt_connected claude = anthropic.Anthropic() matrix = MatrixClient(config.MATRIX_HOMESERVER, config.MATRIX_ACCESS_TOKEN, config.MATRIX_ROOM_ID) loop = asyncio.get_event_loop() def on_connect(client, userdata, flags, reason_code, properties): global mqtt_connected mqtt_connected = True client.subscribe("frigate/events") logger.info("MQTT connected, subscribed to frigate/events") def on_disconnect(client, userdata, flags, reason_code, properties): global mqtt_connected mqtt_connected = False logger.warning("MQTT disconnected: %s", reason_code) def on_message(client, userdata, msg): try: payload = json.loads(msg.payload) asyncio.run_coroutine_threadsafe(process_event(payload, claude, matrix), loop) except json.JSONDecodeError: logger.warning("Invalid JSON on %s", msg.topic) except Exception: logger.exception("Error processing MQTT message") mqtt_client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2) mqtt_client.on_connect = on_connect mqtt_client.on_disconnect = on_disconnect mqtt_client.on_message = on_message def mqtt_loop(): try: mqtt_client.connect(config.MQTT_HOST, config.MQTT_PORT) mqtt_client.loop_forever() except Exception: logger.exception("MQTT connection failed") thread = Thread(target=mqtt_loop, daemon=True) thread.start() # --- Run --- if __name__ == "__main__": import uvicorn uvicorn.run(app, host=config.HOST, port=config.PORT, log_level="info") ``` **Step 7: Update Dockerfile** ```dockerfile FROM python:3.12-slim WORKDIR /app RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/* COPY app/requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app/ . RUN mkdir -p /data EXPOSE 5199 CMD ["python", "-u", "main.py"] ``` **Step 8: Create `tests/conftest.py`** ```python import os import sys import tempfile import pytest # Add app directory to path sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "app")) @pytest.fixture def db_path(): with tempfile.NamedTemporaryFile(suffix=".db", delete=False) as f: path = f.name yield path os.unlink(path) ``` **Step 9: Create `tests/test_database.py`** ```python from database import init_db, get_connection def test_init_db_creates_tables(db_path): init_db(db_path) conn = get_connection(db_path) tables = conn.execute( "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name" ).fetchall() table_names = [t["name"] for t in tables] assert "known_objects" in table_names assert "embeddings" in table_names assert "events" in table_names assert "settings" in table_names def test_default_settings_inserted(db_path): init_db(db_path) conn = get_connection(db_path) row = conn.execute("SELECT value FROM settings WHERE key='vision_backend'").fetchone() assert row["value"] == "haiku" def test_known_object_crud(db_path): init_db(db_path) conn = get_connection(db_path) conn.execute( "INSERT INTO known_objects (name, type) VALUES (?, ?)", ("Jeremy", "person"), ) conn.commit() row = conn.execute("SELECT * FROM known_objects WHERE name='Jeremy'").fetchone() assert row["type"] == "person" assert row["action"] == "quiet" def test_event_insert(db_path): init_db(db_path) conn = get_connection(db_path) conn.execute( """INSERT INTO events (frigate_event_id, camera, label, score, match_status) VALUES (?, ?, ?, ?, ?)""", ("evt-123", "back_porch", "person", 0.92, "unreviewed"), ) conn.commit() row = conn.execute("SELECT * FROM events WHERE frigate_event_id='evt-123'").fetchone() assert row["camera"] == "back_porch" assert row["score"] == 0.92 def test_embedding_cascade_delete(db_path): init_db(db_path) conn = get_connection(db_path) conn.execute("INSERT INTO known_objects (name, type) VALUES ('Dog', 'pet')") obj_id = conn.execute("SELECT last_insert_rowid()").fetchone()[0] conn.execute( "INSERT INTO embeddings (known_object_id, embedding) VALUES (?, ?)", (obj_id, b"fake-embedding"), ) conn.commit() conn.execute("DELETE FROM known_objects WHERE id=?", (obj_id,)) conn.commit() rows = conn.execute("SELECT * FROM embeddings WHERE known_object_id=?", (obj_id,)).fetchall() assert len(rows) == 0 ``` **Step 10: Run tests** ```bash cd /mnt/c/Users/Jeremy/projects/frigate-notify pip install pytest fastapi uvicorn python-multipart numpy paho-mqtt anthropic aiohttp pytest tests/ -v ``` Expected: All 5 tests pass. **Step 11: Commit** ```bash git add -A git commit -m "feat: scaffold v2 with FastAPI, SQLite database, and project structure" git push origin main ``` --- ## Task 2: REST API Endpoints **Goal:** Implement all CRUD API endpoints for events, known objects, and settings. **Files:** - Create: `app/routes/__init__.py` - Create: `app/routes/events.py` - Create: `app/routes/known_objects.py` - Create: `app/routes/settings.py` - Modify: `app/main.py` (include routers) - Create: `tests/test_api.py` **Step 1: Create `app/routes/__init__.py`** ```python # Routes package ``` **Step 2: Create `app/routes/events.py`** ```python import base64 from fastapi import APIRouter, Query, HTTPException from database import get_connection from config import config from models import EventResponse, TagEventRequest router = APIRouter(prefix="/api/events", tags=["events"]) @router.get("") async def list_events( camera: str | None = None, label: str | None = None, status: str | None = None, limit: int = Query(default=50, le=200), offset: int = 0, ): conn = get_connection(config.DB_PATH) query = """ SELECT e.*, ko.name as matched_object_name FROM events e LEFT JOIN known_objects ko ON e.matched_object_id = ko.id WHERE 1=1 """ params = [] if camera: query += " AND e.camera = ?" params.append(camera) if label: query += " AND e.label = ?" params.append(label) if status: query += " AND e.match_status = ?" params.append(status) query += " ORDER BY e.created_at DESC LIMIT ? OFFSET ?" params.extend([limit, offset]) rows = conn.execute(query, params).fetchall() return [ { **{k: row[k] for k in row.keys() if k != "snapshot" and k != "embedding"}, "has_snapshot": row["snapshot"] is not None, "notified": bool(row["notified"]), } for row in rows ] @router.get("/{event_id}") async def get_event(event_id: int): conn = get_connection(config.DB_PATH) row = conn.execute( """SELECT e.*, ko.name as matched_object_name FROM events e LEFT JOIN known_objects ko ON e.matched_object_id = ko.id WHERE e.id = ?""", (event_id,), ).fetchone() if not row: raise HTTPException(404, "Event not found") result = {k: row[k] for k in row.keys() if k != "embedding"} if row["snapshot"]: result["snapshot"] = base64.b64encode(row["snapshot"]).decode() else: result["snapshot"] = None result["notified"] = bool(row["notified"]) return result @router.get("/{event_id}/snapshot") async def get_event_snapshot(event_id: int): from fastapi.responses import Response conn = get_connection(config.DB_PATH) row = conn.execute("SELECT snapshot FROM events WHERE id = ?", (event_id,)).fetchone() if not row or not row["snapshot"]: raise HTTPException(404, "Snapshot not found") return Response(content=row["snapshot"], media_type="image/jpeg") @router.post("/{event_id}/tag") async def tag_event(event_id: int, body: TagEventRequest): conn = get_connection(config.DB_PATH) event = conn.execute("SELECT * FROM events WHERE id = ?", (event_id,)).fetchone() if not event: raise HTTPException(404, "Event not found") if body.known_object_id: obj = conn.execute( "SELECT * FROM known_objects WHERE id = ?", (body.known_object_id,) ).fetchone() if not obj: raise HTTPException(404, "Known object not found") object_id = body.known_object_id elif body.name and body.type: cursor = conn.execute( "INSERT INTO known_objects (name, type) VALUES (?, ?)", (body.name, body.type), ) object_id = cursor.lastrowid else: raise HTTPException(400, "Provide known_object_id or name+type") # Store embedding if event has one if event["embedding"]: conn.execute( """INSERT INTO embeddings (known_object_id, embedding, thumbnail) VALUES (?, ?, ?)""", (object_id, event["embedding"], event["snapshot"]), ) conn.execute( """UPDATE events SET match_status = 'known', matched_object_id = ?, match_confidence = 1.0 WHERE id = ?""", (object_id, event_id), ) conn.commit() return {"success": True, "known_object_id": object_id} @router.post("/{event_id}/dismiss") async def dismiss_event(event_id: int): conn = get_connection(config.DB_PATH) result = conn.execute( "UPDATE events SET match_status = 'unknown' WHERE id = ?", (event_id,) ) conn.commit() if result.rowcount == 0: raise HTTPException(404, "Event not found") return {"success": True} ``` **Step 3: Create `app/routes/known_objects.py`** ```python import base64 from fastapi import APIRouter, HTTPException from database import get_connection from config import config from models import KnownObjectCreate, KnownObjectUpdate router = APIRouter(prefix="/api/known-objects", tags=["known-objects"]) @router.get("") async def list_known_objects(type: str | None = None): conn = get_connection(config.DB_PATH) query = """ SELECT ko.*, COUNT(e.id) as embedding_count FROM known_objects ko LEFT JOIN embeddings e ON ko.id = e.known_object_id """ params = [] if type: query += " WHERE ko.type = ?" params.append(type) query += " GROUP BY ko.id ORDER BY ko.name" rows = conn.execute(query, params).fetchall() return [dict(row) for row in rows] @router.get("/{object_id}") async def get_known_object(object_id: int): conn = get_connection(config.DB_PATH) obj = conn.execute("SELECT * FROM known_objects WHERE id = ?", (object_id,)).fetchone() if not obj: raise HTTPException(404, "Known object not found") embeddings = conn.execute( "SELECT id, created_at FROM embeddings WHERE known_object_id = ?", (object_id,), ).fetchall() result = dict(obj) result["embeddings"] = [dict(e) for e in embeddings] return result @router.get("/{object_id}/thumbnails/{embedding_id}") async def get_thumbnail(object_id: int, embedding_id: int): from fastapi.responses import Response conn = get_connection(config.DB_PATH) row = conn.execute( "SELECT thumbnail FROM embeddings WHERE id = ? AND known_object_id = ?", (embedding_id, object_id), ).fetchone() if not row or not row["thumbnail"]: raise HTTPException(404, "Thumbnail not found") return Response(content=row["thumbnail"], media_type="image/jpeg") @router.put("/{object_id}") async def update_known_object(object_id: int, body: KnownObjectUpdate): conn = get_connection(config.DB_PATH) obj = conn.execute("SELECT * FROM known_objects WHERE id = ?", (object_id,)).fetchone() if not obj: raise HTTPException(404, "Known object not found") updates = {} if body.name is not None: updates["name"] = body.name if body.type is not None: updates["type"] = body.type if body.action is not None: updates["action"] = body.action if updates: set_clause = ", ".join(f"{k} = ?" for k in updates) values = list(updates.values()) + [object_id] conn.execute( f"UPDATE known_objects SET {set_clause}, updated_at = CURRENT_TIMESTAMP WHERE id = ?", values, ) conn.commit() return {"success": True} @router.delete("/{object_id}") async def delete_known_object(object_id: int): conn = get_connection(config.DB_PATH) result = conn.execute("DELETE FROM known_objects WHERE id = ?", (object_id,)) conn.commit() if result.rowcount == 0: raise HTTPException(404, "Known object not found") return {"success": True} ``` **Step 4: Create `app/routes/settings.py`** ```python import json from fastapi import APIRouter from database import get_connection from config import config from models import StatsResponse router = APIRouter(prefix="/api", tags=["settings"]) @router.get("/settings") async def get_settings(): conn = get_connection(config.DB_PATH) rows = conn.execute("SELECT key, value FROM settings").fetchall() result = {row["key"]: row["value"] for row in rows} # Add runtime config values result["match_high_confidence"] = str(config.MATCH_HIGH_CONFIDENCE) result["match_low_confidence"] = str(config.MATCH_LOW_CONFIDENCE) result["rate_limit_seconds"] = str(config.RATE_LIMIT_SECONDS) result["allowed_labels"] = json.dumps(list(config.ALLOWED_LABELS)) result["min_score"] = str(config.MIN_SCORE) return result @router.put("/settings") async def update_settings(body: dict): conn = get_connection(config.DB_PATH) for key, value in body.items(): conn.execute( "INSERT OR REPLACE INTO settings (key, value) VALUES (?, ?)", (key, str(value)), ) conn.commit() return {"success": True} @router.get("/stats") async def get_stats(): conn = get_connection(config.DB_PATH) total = conn.execute("SELECT COUNT(*) as c FROM events").fetchone()["c"] known = conn.execute("SELECT COUNT(*) as c FROM events WHERE match_status='known'").fetchone()["c"] unknown = conn.execute("SELECT COUNT(*) as c FROM events WHERE match_status='unknown'").fetchone()["c"] unreviewed = conn.execute("SELECT COUNT(*) as c FROM events WHERE match_status='unreviewed'").fetchone()["c"] objects = conn.execute("SELECT COUNT(*) as c FROM known_objects").fetchone()["c"] today = conn.execute("SELECT COUNT(*) as c FROM events WHERE date(created_at)=date('now')").fetchone()["c"] return StatsResponse( total_events=total, known_matches=known, unknown_events=unknown, unreviewed=unreviewed, known_objects_count=objects, events_today=today, ) ``` **Step 5: Update `app/main.py` — add router includes after app creation** Add these lines after `app = FastAPI(...)`: ```python from routes.events import router as events_router from routes.known_objects import router as known_objects_router from routes.settings import router as settings_router app.include_router(events_router) app.include_router(known_objects_router) app.include_router(settings_router) ``` **Step 6: Create `tests/test_api.py`** ```python import os import sys import tempfile import pytest from fastapi.testclient import TestClient sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "app")) from database import init_db @pytest.fixture def client(db_path): os.environ["DB_PATH"] = db_path # Must reimport after setting env import importlib import config as config_mod importlib.reload(config_mod) config_mod.config.DB_PATH = db_path init_db(db_path) from main import app return TestClient(app) def test_health(client): resp = client.get("/health") assert resp.status_code == 200 data = resp.json() assert data["status"] == "ok" assert data["version"] == "2.0.0" def test_list_events_empty(client): resp = client.get("/api/events") assert resp.status_code == 200 assert resp.json() == [] def test_stats_empty(client): resp = client.get("/api/stats") assert resp.status_code == 200 data = resp.json() assert data["total_events"] == 0 def test_settings_get(client): resp = client.get("/api/settings") assert resp.status_code == 200 data = resp.json() assert data["vision_backend"] == "haiku" def test_settings_update(client): resp = client.put("/api/settings", json={"vision_backend": "ollama"}) assert resp.status_code == 200 resp = client.get("/api/settings") assert resp.json()["vision_backend"] == "ollama" def test_known_objects_crud(client): # Create via tag from database import get_connection conn = get_connection(client.app.state if hasattr(client.app, 'state') else db_path) # Use the API - create a known object indirectly resp = client.get("/api/known-objects") assert resp.status_code == 200 assert resp.json() == [] def test_event_not_found(client): resp = client.get("/api/events/999") assert resp.status_code == 404 ``` **Step 7: Run tests** ```bash pytest tests/ -v ``` Expected: All tests pass. **Step 8: Commit** ```bash git add -A git commit -m "feat: add REST API endpoints for events, known objects, and settings" git push origin main ``` --- ## Task 3: CLIP Service Sidecar **Goal:** Create a standalone FastAPI service that runs on the GPU machine (192.168.86.196), loads the CLIP model, and provides embedding generation endpoints. **Files:** - Create: `clip-service/app/main.py` - Create: `clip-service/app/requirements.txt` - Create: `clip-service/Dockerfile` - Create: `clip-service/docker-compose.yml` - Create: `clip-service/tests/test_clip.py` **Step 1: Create `clip-service/app/requirements.txt`** ``` fastapi>=0.115.0 uvicorn[standard]>=0.34.0 open-clip-torch>=2.29.0 torch>=2.5.0 Pillow>=10.0.0 numpy>=1.26.0 python-multipart>=0.0.18 ``` **Step 2: Create `clip-service/app/main.py`** ```python import base64 import io import logging import sys import time import numpy as np import open_clip import torch from fastapi import FastAPI, HTTPException from PIL import Image from pydantic import BaseModel logging.basicConfig( level=logging.INFO, format='{"time":"%(asctime)s","level":"%(levelname)s","message":"%(message)s"}', stream=sys.stdout, ) logger = logging.getLogger(__name__) app = FastAPI(title="CLIP Embedding Service", version="1.0.0") # Global model state model = None preprocess = None tokenizer = None device = None class EmbedRequest(BaseModel): image: str # base64-encoded JPEG class EmbedResponse(BaseModel): embedding: list[float] inference_ms: float class SimilarityRequest(BaseModel): embedding1: list[float] embedding2: list[float] class SimilarityResponse(BaseModel): score: float class BatchSimilarityRequest(BaseModel): query_embedding: list[float] candidate_embeddings: list[list[float]] class BatchSimilarityResponse(BaseModel): scores: list[float] best_index: int best_score: float @app.on_event("startup") async def load_model(): global model, preprocess, tokenizer, device device = "cuda" if torch.cuda.is_available() else "cpu" logger.info("Loading CLIP model on %s...", device) model, _, preprocess = open_clip.create_model_and_transforms( "ViT-L-14", pretrained="openai" ) model = model.to(device).eval() tokenizer = open_clip.get_tokenizer("ViT-L-14") logger.info("CLIP model loaded successfully on %s", device) @app.get("/health") async def health(): return { "status": "ok", "device": str(device), "model": "ViT-L-14", } @app.post("/embed", response_model=EmbedResponse) async def embed_image(body: EmbedRequest): try: image_bytes = base64.b64decode(body.image) image = Image.open(io.BytesIO(image_bytes)).convert("RGB") except Exception as e: raise HTTPException(400, f"Invalid image: {e}") start = time.time() image_tensor = preprocess(image).unsqueeze(0).to(device) with torch.no_grad(): embedding = model.encode_image(image_tensor) embedding = embedding / embedding.norm(dim=-1, keepdim=True) elapsed_ms = (time.time() - start) * 1000 embedding_list = embedding.cpu().numpy().flatten().tolist() logger.info("Generated embedding in %.1fms (%d dims)", elapsed_ms, len(embedding_list)) return EmbedResponse(embedding=embedding_list, inference_ms=round(elapsed_ms, 1)) @app.post("/similarity", response_model=SimilarityResponse) async def compute_similarity(body: SimilarityRequest): a = np.array(body.embedding1, dtype=np.float32) b = np.array(body.embedding2, dtype=np.float32) score = float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))) return SimilarityResponse(score=score) @app.post("/batch-similarity", response_model=BatchSimilarityResponse) async def batch_similarity(body: BatchSimilarityRequest): query = np.array(body.query_embedding, dtype=np.float32) candidates = np.array(body.candidate_embeddings, dtype=np.float32) query_norm = query / np.linalg.norm(query) cand_norms = candidates / np.linalg.norm(candidates, axis=1, keepdims=True) scores = (cand_norms @ query_norm).tolist() best_idx = int(np.argmax(scores)) return BatchSimilarityResponse( scores=scores, best_index=best_idx, best_score=scores[best_idx] ) ``` **Step 3: Create `clip-service/Dockerfile`** ```dockerfile FROM pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime WORKDIR /app RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/* COPY app/requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app/ . EXPOSE 5200 CMD ["python", "-u", "-m", "uvicorn", "main:app", "--host", "0.0.0.0", "--port", "5200"] ``` **Step 4: Create `clip-service/docker-compose.yml`** ```yaml services: clip-service: build: context: . dockerfile: Dockerfile container_name: clip-service restart: unless-stopped ports: - "5200:5200" deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] mem_limit: 4G memswap_limit: 4G healthcheck: test: ["CMD-SHELL", "curl -sf http://localhost:5200/health || exit 1"] interval: 30s timeout: 10s retries: 3 start_period: 120s logging: driver: "json-file" options: max-size: "50m" max-file: "3" ``` **Step 5: Create `clip-service/tests/test_clip.py`** ```python import numpy as np def test_cosine_similarity_identical(): """Test that identical embeddings have similarity 1.0""" a = np.random.randn(768).astype(np.float32) a = a / np.linalg.norm(a) score = float(np.dot(a, a)) assert abs(score - 1.0) < 1e-5 def test_cosine_similarity_orthogonal(): """Test that orthogonal embeddings have similarity ~0""" a = np.zeros(768, dtype=np.float32) a[0] = 1.0 b = np.zeros(768, dtype=np.float32) b[1] = 1.0 score = float(np.dot(a, b)) assert abs(score) < 1e-5 def test_batch_similarity(): """Test batch similarity returns correct best match""" query = np.random.randn(768).astype(np.float32) query = query / np.linalg.norm(query) # Candidate 0: random (low similarity) c0 = np.random.randn(768).astype(np.float32) c0 = c0 / np.linalg.norm(c0) # Candidate 1: same as query (high similarity) c1 = query.copy() candidates = np.array([c0, c1]) query_norm = query / np.linalg.norm(query) cand_norms = candidates / np.linalg.norm(candidates, axis=1, keepdims=True) scores = (cand_norms @ query_norm).tolist() best_idx = int(np.argmax(scores)) assert best_idx == 1 assert scores[1] > 0.99 ``` **Step 6: Run CLIP tests locally** ```bash cd /mnt/c/Users/Jeremy/projects/frigate-notify pip install numpy pytest clip-service/tests/ -v ``` Expected: All 3 tests pass (these are pure numpy tests, no GPU needed). **Step 7: Commit** ```bash git add clip-service/ git commit -m "feat: add CLIP embedding service sidecar for GPU machine" git push origin main ``` --- ## Task 4: Recognition Engine **Goal:** Add the recognition pipeline that uses CLIP embeddings to match events against known objects, with vision LLM fallback for ambiguous cases. **Files:** - Create: `app/recognition.py` - Create: `app/ollama_client.py` - Modify: `app/vision.py` (add Ollama support) - Create: `tests/test_recognition.py` **Step 1: Create `app/ollama_client.py`** ```python import base64 import logging import aiohttp logger = logging.getLogger(__name__) class OllamaClient: def __init__(self, base_url: str, model: str): self.base_url = base_url.rstrip("/") self.model = model async def analyze_image(self, image_bytes: bytes, prompt: str) -> str | None: """Send image to Ollama vision model for analysis.""" b64 = base64.b64encode(image_bytes).decode("utf-8") payload = { "model": self.model, "prompt": prompt, "images": [b64], "stream": False, "options": {"num_predict": 200}, } try: async with aiohttp.ClientSession() as session: async with session.post( f"{self.base_url}/api/generate", json=payload, timeout=aiohttp.ClientTimeout(total=30), ) as resp: if resp.status == 200: data = await resp.json() return data.get("response", "").strip() logger.warning("Ollama returned %d", resp.status) except Exception: logger.exception("Ollama request failed") return None async def compare_images( self, image_bytes: bytes, reference_bytes: bytes, object_name: str ) -> bool | None: """Ask vision LLM if two images show the same person/vehicle/pet.""" b64_current = base64.b64encode(image_bytes).decode("utf-8") b64_ref = base64.b64encode(reference_bytes).decode("utf-8") prompt = ( f"Look at these two security camera images. The first is a new detection. " f"The second is a reference image of '{object_name}'. " f"Are they the same person/vehicle/animal? " f"Answer ONLY 'yes' or 'no'." ) payload = { "model": self.model, "prompt": prompt, "images": [b64_current, b64_ref], "stream": False, "options": {"num_predict": 10}, } try: async with aiohttp.ClientSession() as session: async with session.post( f"{self.base_url}/api/generate", json=payload, timeout=aiohttp.ClientTimeout(total=30), ) as resp: if resp.status == 200: data = await resp.json() answer = data.get("response", "").strip().lower() if "yes" in answer: return True if "no" in answer: return False logger.warning("Ollama compare returned %d", resp.status) except Exception: logger.exception("Ollama compare failed") return None async def is_available(self) -> bool: try: async with aiohttp.ClientSession() as session: async with session.get( f"{self.base_url}/api/tags", timeout=aiohttp.ClientTimeout(total=3), ) as resp: return resp.status == 200 except Exception: return False ``` **Step 2: Create `app/recognition.py`** ```python import base64 import logging import struct import aiohttp import numpy as np from config import config from database import get_connection logger = logging.getLogger(__name__) def embedding_to_bytes(embedding: list[float]) -> bytes: return struct.pack(f"{len(embedding)}f", *embedding) def bytes_to_embedding(data: bytes) -> np.ndarray: count = len(data) // 4 return np.array(struct.unpack(f"{count}f", data), dtype=np.float32) async def get_clip_embedding(image_bytes: bytes) -> list[float] | None: """Get CLIP embedding from the clip-service sidecar.""" b64 = base64.b64encode(image_bytes).decode("utf-8") try: async with aiohttp.ClientSession() as session: async with session.post( f"{config.CLIP_SERVICE_URL}/embed", json={"image": b64}, timeout=aiohttp.ClientTimeout(total=10), ) as resp: if resp.status == 200: data = await resp.json() logger.info("CLIP embedding: %.1fms", data["inference_ms"]) return data["embedding"] logger.warning("CLIP service returned %d", resp.status) except Exception: logger.exception("CLIP service unavailable") return None async def find_best_match( query_embedding: list[float], ) -> tuple[int | None, float, str | None]: """Compare query embedding against all stored embeddings. Returns (known_object_id, best_score, object_name) or (None, 0.0, None). """ conn = get_connection(config.DB_PATH) rows = conn.execute( """SELECT e.known_object_id, e.embedding, ko.name FROM embeddings e JOIN known_objects ko ON e.known_object_id = ko.id""" ).fetchall() if not rows: return None, 0.0, None query = np.array(query_embedding, dtype=np.float32) query = query / np.linalg.norm(query) best_score = 0.0 best_object_id = None best_name = None # Group by object, take max score per object object_scores: dict[int, tuple[float, str]] = {} for row in rows: candidate = bytes_to_embedding(row["embedding"]) candidate = candidate / np.linalg.norm(candidate) score = float(np.dot(query, candidate)) obj_id = row["known_object_id"] if obj_id not in object_scores or score > object_scores[obj_id][0]: object_scores[obj_id] = (score, row["name"]) if object_scores: best_object_id = max(object_scores, key=lambda k: object_scores[k][0]) best_score, best_name = object_scores[best_object_id] return best_object_id, best_score, best_name async def recognize_event( image_bytes: bytes, ollama_client=None ) -> tuple[str, int | None, float, str | None]: """Run the full recognition pipeline. Returns (match_status, matched_object_id, confidence, object_name). """ # Step 1: Get CLIP embedding embedding = await get_clip_embedding(image_bytes) if embedding is None: return "unreviewed", None, 0.0, None # Step 2: Find best match object_id, score, name = await find_best_match(embedding) if object_id is None: return "unknown", None, 0.0, None # Step 3: Apply confidence thresholds if score >= config.MATCH_HIGH_CONFIDENCE: logger.info("High confidence match: %s (%.2f)", name, score) return "known", object_id, score, name if score < config.MATCH_LOW_CONFIDENCE: logger.info("Low confidence, treating as unknown (%.2f)", score) return "unknown", None, score, None # Step 4: Ambiguous — ask vision LLM logger.info("Ambiguous match: %s (%.2f), consulting vision LLM", name, score) if ollama_client: # Get a reference thumbnail for this object conn = get_connection(config.DB_PATH) ref = conn.execute( "SELECT thumbnail FROM embeddings WHERE known_object_id = ? AND thumbnail IS NOT NULL LIMIT 1", (object_id,), ).fetchone() if ref and ref["thumbnail"]: result = await ollama_client.compare_images( image_bytes, ref["thumbnail"], name ) if result is True: logger.info("Vision LLM confirmed: %s", name) return "known", object_id, score, name elif result is False: logger.info("Vision LLM rejected: %s", name) return "unknown", None, score, None # LLM unavailable or inconclusive — mark ambiguous for human review return "ambiguous", object_id, score, name ``` **Step 3: Modify `app/vision.py` — add Ollama support** ```python import base64 import logging import anthropic logger = logging.getLogger(__name__) PROMPT = ( "Briefly describe what's happening in this security camera image in 1-2 sentences. " "Focus on the people, vehicles, or animals and their activity. Be concise." ) def analyze_snapshot(client: anthropic.Anthropic, image_bytes: bytes) -> str | None: """Send a snapshot to Claude Haiku for description.""" try: b64 = base64.standard_b64encode(image_bytes).decode("utf-8") response = client.messages.create( model="claude-haiku-4-5-20251001", max_tokens=150, messages=[ { "role": "user", "content": [ { "type": "image", "source": { "type": "base64", "media_type": "image/jpeg", "data": b64, }, }, {"type": "text", "text": PROMPT}, ], } ], ) description = response.content[0].text.strip() logger.info("Haiku analysis complete: %d chars", len(description)) return description except Exception: logger.exception("Claude Vision API failed") return None async def analyze_snapshot_ollama(ollama_client, image_bytes: bytes) -> str | None: """Send a snapshot to Ollama vision model for description.""" return await ollama_client.analyze_image(image_bytes, PROMPT) async def get_description( image_bytes: bytes, vision_backend: str, claude_client=None, ollama_client=None, ) -> str | None: """Get scene description using configured backend.""" if vision_backend == "ollama" and ollama_client: result = await analyze_snapshot_ollama(ollama_client, image_bytes) if result: return result # Fall through to haiku if ollama fails if claude_client: return analyze_snapshot(claude_client, image_bytes) elif vision_backend == "haiku" and claude_client: return analyze_snapshot(claude_client, image_bytes) elif vision_backend == "both" and ollama_client: result = await analyze_snapshot_ollama(ollama_client, image_bytes) if result: return result if claude_client: return analyze_snapshot(claude_client, image_bytes) elif claude_client: return analyze_snapshot(claude_client, image_bytes) return None ``` **Step 4: Create `tests/test_recognition.py`** ```python import struct import numpy as np from recognition import embedding_to_bytes, bytes_to_embedding def test_embedding_roundtrip(): original = np.random.randn(768).astype(np.float32).tolist() blob = embedding_to_bytes(original) recovered = bytes_to_embedding(blob) np.testing.assert_array_almost_equal(recovered, original, decimal=5) def test_embedding_to_bytes_length(): embedding = [0.0] * 768 blob = embedding_to_bytes(embedding) assert len(blob) == 768 * 4 # float32 = 4 bytes each def test_bytes_to_embedding_shape(): blob = struct.pack("768f", *([0.0] * 768)) result = bytes_to_embedding(blob) assert result.shape == (768,) assert result.dtype == np.float32 ``` **Step 5: Run tests** ```bash pytest tests/ -v ``` Expected: All tests pass. **Step 6: Commit** ```bash git add -A git commit -m "feat: add recognition engine with CLIP matching and Ollama vision LLM" git push origin main ``` --- ## Task 5: Integrate Recognition into Event Pipeline **Goal:** Wire the recognition engine into the MQTT event processing flow so events are automatically matched against known objects before deciding whether to send Matrix alerts. **Files:** - Modify: `app/main.py` (integrate recognition into process_event) - Modify: `app/docker-compose.yml` (add CLIP/Ollama env vars) - Create: `tests/test_event_flow.py` **Step 1: Update `app/main.py` process_event function** Replace the existing `process_event` function body (after snapshot fetch) with recognition-aware logic. The key change: before sending a Matrix notification, check if recognition is enabled in settings, run the recognition pipeline, and only notify for unknown objects. Add imports at top of main.py: ```python from recognition import recognize_event, get_clip_embedding, embedding_to_bytes from ollama_client import OllamaClient from vision import get_description ``` Add ollama_client initialization in startup: ```python ollama_client = OllamaClient(config.OLLAMA_URL, config.OLLAMA_VISION_MODEL) ``` Update process_event to accept and use ollama_client, and add recognition logic between snapshot fetch and notification: ```python # After fetching snapshot and storing event... # Check if recognition is enabled recognition_enabled = False try: row = conn.execute("SELECT value FROM settings WHERE key='recognition_enabled'").fetchone() if row: recognition_enabled = row["value"].lower() == "true" except Exception: pass # Run recognition if enabled match_status = "unreviewed" matched_id = None match_conf = 0.0 matched_name = None if recognition_enabled and snapshot: match_status, matched_id, match_conf, matched_name = await recognize_event( snapshot, ollama_client ) # Store embedding embedding = await get_clip_embedding(snapshot) if embedding: embedding_blob = embedding_to_bytes(embedding) try: conn.execute( "UPDATE events SET embedding=?, match_status=?, matched_object_id=?, match_confidence=? WHERE frigate_event_id=?", (embedding_blob, match_status, matched_id, match_conf, event_id), ) conn.commit() except Exception: logger.exception("Failed to update event recognition") # If known, check action if match_status == "known" and matched_id: obj = conn.execute("SELECT action FROM known_objects WHERE id=?", (matched_id,)).fetchone() if obj and obj["action"] in ("quiet", "suppress"): logger.info("Known object '%s' (%s), skipping notification", matched_name, obj["action"]) events_processed += 1 last_event_time = time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()) return # Get AI description using configured backend vision_backend = "haiku" try: row = conn.execute("SELECT value FROM settings WHERE key='vision_backend'").fetchone() if row: vision_backend = row["value"] except Exception: pass description = await get_description(snapshot, vision_backend, claude, ollama_client) if snapshot else None if not description: description = f"{label.capitalize()} detected on {camera}." # ... rest of notification code unchanged ... ``` **Step 2: Update `docker-compose.yml`** ```yaml services: frigate-notify: build: context: . dockerfile: Dockerfile container_name: frigate-notify restart: unless-stopped network_mode: host environment: - TZ=America/Chicago - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} - MATRIX_ACCESS_TOKEN=${MATRIX_ACCESS_TOKEN} - MATRIX_ROOM_ID=${MATRIX_ROOM_ID} - FRIGATE_URL=http://192.168.86.192:5000 - MQTT_HOST=192.168.86.192 - MQTT_PORT=1883 - MATRIX_HOMESERVER=http://192.168.86.167:8008 - CLIP_SERVICE_URL=http://192.168.86.196:5200 - OLLAMA_URL=http://192.168.86.196:11434 - OLLAMA_VISION_MODEL=llava:13b - DB_PATH=/data/frigate-notify.db volumes: - /mnt/docker-data/configs/frigate-notify:/data mem_limit: 256M memswap_limit: 256M healthcheck: test: ["CMD-SHELL", "curl -sf http://localhost:5199/health || exit 1"] interval: 30s timeout: 10s retries: 3 start_period: 30s logging: driver: "json-file" options: max-size: "50m" max-file: "3" ``` **Step 3: Commit** ```bash git add -A git commit -m "feat: integrate recognition engine into MQTT event pipeline" git push origin main ``` --- ## Task 6: Web UI — Dashboard & Event Review **Goal:** Create the vanilla HTML/CSS/JS web interface with Dashboard and Review Queue pages. **Files:** - Create: `app/static/index.html` - Create: `app/static/style.css` - Create: `app/static/app.js` **Step 1: Create `app/static/index.html`** ```html Frigate Alerts
``` **Step 2: Create `app/static/style.css`** ```css * { margin: 0; padding: 0; box-sizing: border-box; } body { font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; background: #1a1a2e; color: #e0e0e0; display: flex; min-height: 100vh; } #sidebar { width: 220px; background: #16213e; padding: 16px 0; flex-shrink: 0; border-right: 1px solid #0f3460; } .sidebar-header { padding: 0 16px 16px; border-bottom: 1px solid #0f3460; } .sidebar-header h2 { font-size: 18px; color: #e94560; } .sidebar-section { padding: 8px 0; } .sidebar-item { display: block; padding: 10px 16px; color: #a0a0b0; text-decoration: none; font-size: 14px; transition: background 0.2s; } .sidebar-item:hover { background: #1a1a40; color: #fff; } .sidebar-item.active { background: #0f3460; color: #e94560; font-weight: 600; } .sidebar-stats { padding: 16px; font-size: 12px; color: #808090; } #content { flex: 1; padding: 24px; overflow-y: auto; } .page-title { font-size: 24px; margin-bottom: 16px; } .filters { display: flex; gap: 12px; margin-bottom: 16px; flex-wrap: wrap; } .filters select, .filters input { background: #16213e; color: #e0e0e0; border: 1px solid #0f3460; padding: 6px 10px; border-radius: 4px; font-size: 13px; } .event-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(300px, 1fr)); gap: 16px; } .event-card { background: #16213e; border: 1px solid #0f3460; border-radius: 8px; overflow: hidden; transition: border-color 0.2s; } .event-card:hover { border-color: #e94560; } .event-card img { width: 100%; height: 180px; object-fit: cover; background: #0a0a1a; } .event-card .event-info { padding: 12px; } .event-card .event-header { display: flex; justify-content: space-between; margin-bottom: 6px; } .event-card .event-camera { font-weight: 600; font-size: 14px; } .event-card .event-label { font-size: 12px; padding: 2px 8px; border-radius: 12px; background: #0f3460; color: #e0e0e0; } .event-card .event-description { font-size: 13px; color: #a0a0b0; margin: 6px 0; } .event-card .event-meta { font-size: 11px; color: #606070; } .event-card .event-actions { display: flex; gap: 8px; margin-top: 8px; } .status-known { border-left: 3px solid #4caf50; } .status-unknown { border-left: 3px solid #e94560; } .status-ambiguous { border-left: 3px solid #ff9800; } .status-unreviewed { border-left: 3px solid #2196f3; } .btn { padding: 6px 14px; border: none; border-radius: 4px; cursor: pointer; font-size: 13px; transition: background 0.2s; } .btn-primary { background: #e94560; color: #fff; } .btn-primary:hover { background: #c73450; } .btn-success { background: #4caf50; color: #fff; } .btn-success:hover { background: #3d9140; } .btn-danger { background: #f44336; color: #fff; } .btn { background: #0f3460; color: #e0e0e0; } .btn:hover { background: #1a4a7a; } .known-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(250px, 1fr)); gap: 16px; } .known-card { background: #16213e; border: 1px solid #0f3460; border-radius: 8px; padding: 16px; } .known-card h4 { margin-bottom: 8px; } .known-card .known-type { font-size: 12px; color: #a0a0b0; margin-bottom: 8px; } .known-card .known-thumbs { display: flex; gap: 4px; flex-wrap: wrap; margin: 8px 0; } .known-card .known-thumbs img { width: 50px; height: 50px; object-fit: cover; border-radius: 4px; } .settings-form { max-width: 500px; } .form-group { margin-bottom: 16px; } .form-group label { display: block; margin-bottom: 4px; font-size: 13px; color: #a0a0b0; } .form-group select, .form-group input { width: 100%; background: #16213e; color: #e0e0e0; border: 1px solid #0f3460; padding: 8px 10px; border-radius: 4px; font-size: 14px; } .modal { position: fixed; inset: 0; background: rgba(0,0,0,0.7); display: flex; align-items: center; justify-content: center; z-index: 100; } .modal.hidden { display: none; } .modal-content { background: #16213e; border-radius: 8px; padding: 24px; min-width: 400px; max-width: 500px; } .modal-content h3 { margin-bottom: 16px; } .modal-content img { width: 100%; max-height: 200px; object-fit: cover; border-radius: 4px; margin-bottom: 12px; } .modal-actions { display: flex; gap: 8px; justify-content: flex-end; margin-top: 16px; } .empty-state { text-align: center; padding: 60px 20px; color: #606070; } .empty-state h3 { margin-bottom: 8px; color: #a0a0b0; } ``` **Step 3: Create `app/static/app.js`** This is the main UI logic. It handles navigation, data fetching, rendering, and the tag modal. The file will be ~400 lines. Key sections: ```javascript let currentView = "dashboard"; let eventsData = []; let knownObjects = []; let stats = {}; // --- Navigation --- document.querySelectorAll(".sidebar-item").forEach(item => { item.addEventListener("click", e => { e.preventDefault(); currentView = item.dataset.view; document.querySelectorAll(".sidebar-item").forEach(i => i.classList.remove("active")); item.classList.add("active"); renderMain(); }); }); // --- Data fetching --- async function fetchEvents(params = {}) { const query = new URLSearchParams(params).toString(); const resp = await fetch(`/api/events?${query}`); return resp.json(); } async function fetchEvent(id) { const resp = await fetch(`/api/events/${id}`); return resp.json(); } async function fetchKnownObjects() { const resp = await fetch("/api/known-objects"); return resp.json(); } async function fetchStats() { const resp = await fetch("/api/stats"); return resp.json(); } async function fetchSettings() { const resp = await fetch("/api/settings"); return resp.json(); } // --- Rendering --- async function renderMain() { const container = document.getElementById("main-content"); stats = await fetchStats(); renderStats(); switch (currentView) { case "dashboard": await renderDashboard(container); break; case "review": await renderReview(container); break; case "known": await renderKnown(container); break; case "settings": await renderSettings(container); break; } } function renderStats() { document.getElementById("stats-summary").innerHTML = `
Today: ${stats.events_today} events
Known: ${stats.known_matches}
Unknown: ${stats.unknown_events}
Review: ${stats.unreviewed}
`; } async function renderDashboard(container) { eventsData = await fetchEvents({ limit: 50 }); container.innerHTML = `

Dashboard

`; renderEventGrid(eventsData); } function renderEventGrid(events) { const grid = document.getElementById("event-grid"); if (!events.length) { grid.innerHTML = '

No events

Events will appear here when Frigate detects objects.

'; return; } grid.innerHTML = events.map(e => `
${e.has_snapshot ? `${e.camera}` : '
No snapshot
'}
${e.camera} ${e.label} ${Math.round(e.score * 100)}%
${e.ai_description || ""}
${e.matched_object_name ? `
Matched: ${e.matched_object_name} (${Math.round((e.match_confidence || 0) * 100)}%)
` : ""}
${new Date(e.created_at).toLocaleString()}
`).join(""); } async function applyFilters() { const camera = document.getElementById("filter-camera").value; const status = document.getElementById("filter-status").value; const params = { limit: 50 }; if (camera) params.camera = camera; if (status) params.status = status; const filtered = await fetchEvents(params); renderEventGrid(filtered); } async function renderReview(container) { const events = await fetchEvents({ status: "unreviewed", limit: 50 }); container.innerHTML = `

Review Queue (${events.length})

`; renderEventGrid(events); } async function renderKnown(container) { knownObjects = await fetchKnownObjects(); container.innerHTML = `

Known Objects (${knownObjects.length})

${knownObjects.length === 0 ? '

No known objects

Tag events from the Dashboard or Review Queue to add objects here.

' : knownObjects.map(obj => `

${obj.name}

${obj.type} · ${obj.embedding_count} samples
`).join("")}
`; } async function renderSettings(container) { const settings = await fetchSettings(); container.innerHTML = `

Settings

`; } // --- Actions --- async function dismissEvent(id) { await fetch(`/api/events/${id}/dismiss`, { method: "POST" }); renderMain(); } async function deleteKnownObject(id) { if (!confirm("Delete this known object and all its reference data?")) return; await fetch(`/api/known-objects/${id}`, { method: "DELETE" }); renderMain(); } async function saveSetting(key, value) { await fetch("/api/settings", { method: "PUT", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ [key]: value }), }); } // --- Tag Modal --- let tagEventId = null; async function openTagModal(eventId) { tagEventId = eventId; const modal = document.getElementById("tag-modal"); const snapshotDiv = document.getElementById("tag-modal-snapshot"); snapshotDiv.innerHTML = ``; // Populate existing objects dropdown const objects = await fetchKnownObjects(); const select = document.getElementById("tag-existing-object"); select.innerHTML = '' + objects.map(o => ``).join(""); select.onchange = () => { document.getElementById("tag-new-fields").style.display = select.value ? "none" : "block"; }; document.getElementById("tag-new-fields").style.display = "block"; modal.classList.remove("hidden"); } document.getElementById("tag-cancel").addEventListener("click", () => { document.getElementById("tag-modal").classList.add("hidden"); }); document.getElementById("tag-save").addEventListener("click", async () => { const existingId = document.getElementById("tag-existing-object").value; let body; if (existingId) { body = { known_object_id: parseInt(existingId) }; } else { const name = document.getElementById("tag-name").value.trim(); const type = document.getElementById("tag-type").value; if (!name) { alert("Please enter a name"); return; } body = { name, type }; } await fetch(`/api/events/${tagEventId}/tag`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body), }); document.getElementById("tag-modal").classList.add("hidden"); renderMain(); }); // --- Auto-refresh --- setInterval(() => { if (currentView === "dashboard") renderMain(); }, 30000); // --- Init --- renderMain(); ``` **Step 4: Commit** ```bash git add app/static/ git commit -m "feat: add web UI with dashboard, review queue, known objects, and settings pages" git push origin main ``` --- ## Task 7: Event Retention & Cleanup **Goal:** Add automatic cleanup of old events based on the retention setting. **Files:** - Create: `app/cleanup.py` - Modify: `app/main.py` (schedule cleanup on startup) - Create: `tests/test_cleanup.py` **Step 1: Create `app/cleanup.py`** ```python import asyncio import logging from database import get_connection from config import config logger = logging.getLogger(__name__) async def cleanup_old_events(): """Delete events older than the configured retention period.""" while True: try: conn = get_connection(config.DB_PATH) row = conn.execute( "SELECT value FROM settings WHERE key='event_retention_days'" ).fetchone() days = int(row["value"]) if row else 90 result = conn.execute( "DELETE FROM events WHERE created_at < datetime('now', ? || ' days')", (f"-{days}",), ) conn.commit() if result.rowcount > 0: logger.info("Cleaned up %d events older than %d days", result.rowcount, days) except Exception: logger.exception("Cleanup failed") await asyncio.sleep(3600) # Run every hour ``` **Step 2: Add to `app/main.py` startup** ```python from cleanup import cleanup_old_events @app.on_event("startup") async def startup(): init_db(config.DB_PATH) start_mqtt() asyncio.create_task(cleanup_old_events()) logger.info("Frigate-notify v2 started") ``` **Step 3: Create `tests/test_cleanup.py`** ```python import sqlite3 from database import init_db def test_cleanup_deletes_old_events(db_path): init_db(db_path) conn = sqlite3.connect(db_path) conn.row_factory = sqlite3.Row # Insert an old event conn.execute( """INSERT INTO events (frigate_event_id, camera, label, score, match_status, created_at) VALUES ('old-1', 'cam', 'person', 0.9, 'unknown', datetime('now', '-100 days'))""" ) # Insert a recent event conn.execute( """INSERT INTO events (frigate_event_id, camera, label, score, match_status) VALUES ('new-1', 'cam', 'person', 0.9, 'unknown')""" ) conn.commit() result = conn.execute( "DELETE FROM events WHERE created_at < datetime('now', '-90 days')" ) conn.commit() assert result.rowcount == 1 remaining = conn.execute("SELECT COUNT(*) as c FROM events").fetchone()["c"] assert remaining == 1 conn.close() ``` **Step 4: Run tests and commit** ```bash pytest tests/ -v git add -A git commit -m "feat: add automatic event cleanup based on retention setting" git push origin main ``` --- ## Task 8: Farm Manager Sidebar Link **Goal:** Add a "Frigate Alerts" link to the Farm Manager "Server Links" page. **Files:** - Modify: `/mnt/c/Users/Jeremy/projects/farm-manager/server/static/app.js` (add link to serverLinks array) **Step 1: Add link entry** In `app.js` around line 734, the `serverLinks` array has categories. Add to the appropriate node category (hf-pdocker-01 section): ```javascript { name: "Frigate Alerts", url: "http://192.168.86.192:5199", icon: "\uD83D\uDD14" }, ``` This adds a bell icon link to the frigate-notify v2 web UI. **Step 2: Commit in the farm-manager repo** ```bash cd /mnt/c/Users/Jeremy/projects/farm-manager git add server/static/app.js git commit -m "feat: add Frigate Alerts link to server links page" git push origin main ``` --- ## Task 9: Pull Ollama Vision Model **Goal:** Pull `llava:13b` on the GPU machine so vision LLM is ready. **Step 1: Pull the model** ```bash curl -X POST http://192.168.86.196:11434/api/pull -d '{"name":"llava:13b"}' ``` This will take several minutes to download (~8GB). Verify with: ```bash curl http://192.168.86.196:11434/api/tags | python3 -m json.tool ``` Expected: `llava:13b` appears in the model list. --- ## Task 10: Deploy clip-service to GPU Machine **Goal:** Build and deploy the CLIP service container on 192.168.86.196. **Step 1: Copy clip-service files to GPU machine** ```bash scp -r /mnt/c/Users/Jeremy/projects/frigate-notify/clip-service/ glyptic@192.168.86.196:/tmp/clip-service/ ``` **Step 2: Build and start on GPU machine** ```bash ssh glyptic@192.168.86.196 cd /tmp/clip-service docker compose up -d --build ``` **Step 3: Verify** ```bash curl http://192.168.86.196:5200/health ``` Expected: `{"status":"ok","device":"cuda","model":"ViT-L-14"}` **Step 4: Test embedding generation** ```bash # Create a small test image (1x1 pixel JPEG) python3 -c " import base64, json # Use a real test snapshot if available, or a minimal JPEG with open('/path/to/test.jpg', 'rb') as f: b64 = base64.b64encode(f.read()).decode() import urllib.request req = urllib.request.Request( 'http://192.168.86.196:5200/embed', data=json.dumps({'image': b64}).encode(), headers={'Content-Type': 'application/json'} ) resp = urllib.request.urlopen(req) data = json.loads(resp.read()) print(f'Embedding dims: {len(data[\"embedding\"])}, time: {data[\"inference_ms\"]}ms') " ``` Expected: 768-dimensional embedding, <100ms inference time. --- ## Task 11: Deploy frigate-notify v2 **Goal:** Build and deploy the updated frigate-notify on hf-pdocker-01 (when it's back online). **NOTE:** This task is blocked until hf-pdocker-01 is restored. The compose file and code are ready on the NAS. **Step 1: Create data directory on NAS** ```bash ssh glyptic@192.168.86.100 mkdir -p /mnt/docker-data/configs/frigate-notify ``` **Step 2: Copy updated code to NAS** ```bash # From local machine, sync to NAS via any node scp -r /mnt/c/Users/Jeremy/projects/frigate-notify/{app,Dockerfile,docker-compose.yml} \ glyptic@192.168.86.100:/mnt/docker-data/compose/hf-pdocker-01/frigate-notify/ ``` **Step 3: Deploy on hf-pdocker-01 (when available)** ```bash ssh glyptic@192.168.86.192 cd /mnt/docker-data/compose/hf-pdocker-01/frigate-notify docker compose up -d --build ``` **Step 4: Verify** ```bash curl http://192.168.86.192:5199/health # Open http://192.168.86.192:5199 in browser — should show Dashboard ``` --- ## Summary | Task | Component | Dependencies | |------|-----------|-------------| | 1 | Project scaffold, FastAPI, SQLite | None | | 2 | REST API endpoints | Task 1 | | 3 | CLIP service sidecar | None (independent) | | 4 | Recognition engine | Task 1 | | 5 | Pipeline integration | Tasks 2, 4 | | 6 | Web UI | Task 2 | | 7 | Event cleanup | Task 1 | | 8 | Farm Manager link | None (independent) | | 9 | Pull Ollama model | None (independent) | | 10 | Deploy clip-service | Task 3 | | 11 | Deploy frigate-notify v2 | Tasks 1-7, hf-pdocker-01 online | **Independent tasks (can run in parallel):** 1, 3, 8, 9 **Sequential chain:** 1 → 2 → 4 → 5 → 6 → 7 → 11