hobbs-iac/docs/plans/2026-03-07-frigate-notify-v2-design.md

# Frigate-Notify v2 Design

## Summary

Evolve frigate-notify from a simple MQTT-to-Matrix alert forwarder into an intelligent notification system with known-object recognition, a web UI for managing identities and reviewing events, and configurable vision backends (local Ollama on a 3090 GPU or Claude Haiku cloud API).

## Goals

1. **Reduce noise** — stop alerting on known family members, vehicles, and pets
2. **Learn from real events** — tag objects from Frigate detection snapshots to build a recognition database over time
3. **Local AI processing** — use a local 3090 GPU (192.168.86.196) for CLIP embeddings and vision LLM, reducing cloud API costs
4. **Web interface** — standalone app for managing known objects, reviewing events, and configuring settings
5. **Farm Manager integration** — sidebar link in Farm Manager pointing to the frigate-notify UI

## Architecture

```
Frigate NVR ──MQTT──> Mosquitto Broker
                           |
                           v
                    frigate-notify v2 (:5199)
                    +-------------------------+
                    |  MQTT Listener           |
                    |  Event Processor         |
                    |  Web UI (HTML/CSS/JS)    |
                    |  REST API (FastAPI)      |
                    |  SQLite DB               |
                    +------+------+------+-----+
                           |      |      |
              +------------+      |      +------------+
              v                   v                    v
    clip-service (:5200)    Ollama (:11434)      Claude Haiku API
    192.168.86.196          192.168.86.196        cloud
    CLIP embeddings         Vision LLM            Vision fallback
                            Scene descriptions    Scene descriptions
              |                   |
              +-------+-----------+
                      v
              GPU Machine (RTX 3090, 24GB VRAM)

                    frigate-notify v2
                           |
                           v
                    Matrix API
                    #video-alerts room
                    (unknown objects only)

Farm Manager (:8888) ──sidebar link──> frigate-notify (:5199)
```

## Known Object Recognition Pipeline

Each Frigate event goes through this flow:

```
Event arrives (MQTT, type="end")
  -> Fetch snapshot from Frigate API
  -> Generate CLIP embedding via clip-service (192.168.86.196:5200)
  -> Compare against stored embeddings in SQLite
      -> High confidence match (>0.85): "known" -- log quietly, no alert
      -> Low confidence (<0.60): "unknown" -- send Matrix alert
      -> Ambiguous (0.60-0.85): send snapshot + reference images to Vision LLM
          -> LLM confirms match: "known" -- log quietly
          -> LLM says no match: "unknown" -- send Matrix alert
```

Scene description (the AI-generated text in notifications) is handled separately and is configurable to use either Haiku or Ollama vision model.

### Fallback Chain

1. clip-service down: skip embedding matching, send all events to vision LLM
2. Ollama down: fall back to Claude Haiku for everything
3. Haiku down: structured text description ("Person detected on back_porch")

### Known Object Types

- **People** -- tagged by name, matched by face/body appearance
- **Vehicles** -- tagged by description (e.g., "blue F-150"), matched by shape/color
- **Pets** -- tagged by name, matched by appearance

### Learning Flow

1. Events stream in and show up in the web UI's "Recent Events" feed
2. User taps an event and chooses "Tag as known" -> enters name/description -> selects type (person/vehicle/pet)
3. System stores the CLIP embedding from that snapshot tied to that identity
4. Multiple tagged snapshots of the same identity improve matching accuracy over time
5. User can also "Confirm" or "Reject" the system's auto-matches to refine it

### Notification Behavior for Known Objects

- **Known objects**: logged quietly in the web UI, no Matrix notification
- **Unknown objects**: full Matrix alert with snapshot + AI description (existing behavior)

## Web UI

Served at `:5199`, vanilla HTML/CSS/JS (consistent with Farm Manager). Same container as the API and event processor.

### Pages

| Page | Purpose |
|------|---------|
| Dashboard | Live feed of recent events. Each shows snapshot thumbnail, camera, label, score, known/unknown status, AI description. Filter by camera, label, status. |
| Known Objects | Grid of all tagged identities. Click to see all associated snapshots, edit name, delete. Organized by type (people/vehicles/pets). |
| Review Queue | Unreviewed events needing tagging. Quick-action buttons: "Tag as known", "Dismiss", "Confirm match". |
| Settings | Vision backend toggle (Haiku/Ollama/both), confidence thresholds, rate limiting, camera enable/disable, Matrix notification preferences. |

## REST API

| Endpoint | Method | Purpose |
|----------|--------|---------|
| `/api/events` | GET | List recent events, filterable |
| `/api/events/{id}` | GET | Single event detail |
| `/api/events/{id}/tag` | POST | Tag event as known object |
| `/api/events/{id}/dismiss` | POST | Dismiss event |
| `/api/known-objects` | GET | List all known identities |
| `/api/known-objects/{id}` | GET/PUT/DELETE | Manage a known identity |
| `/api/settings` | GET/PUT | Read/update configuration |
| `/api/stats` | GET | Event counts, match rates |
| `/health` | GET | Existing health endpoint (preserved) |

## Data Model

SQLite database at `/mnt/docker-data/configs/frigate-notify/frigate-notify.db`

### Tables

```sql
known_objects
  id              INTEGER PRIMARY KEY
  name            TEXT NOT NULL        -- "Jeremy", "Blue F-150", "Max the dog"
  type            TEXT NOT NULL        -- "person", "vehicle", "pet"
  action          TEXT DEFAULT "quiet" -- "quiet" (log only) | "suppress" | "alert"
  created_at      TIMESTAMP
  updated_at      TIMESTAMP

embeddings
  id              INTEGER PRIMARY KEY
  known_object_id INTEGER FK -> known_objects
  embedding       BLOB                -- CLIP vector (float32 array)
  snapshot_url    TEXT                 -- Frigate snapshot URL for reference
  thumbnail       BLOB                -- Small JPEG for UI display
  created_at      TIMESTAMP

events
  id              INTEGER PRIMARY KEY
  frigate_event_id TEXT UNIQUE         -- Frigate's event ID
  camera          TEXT
  label           TEXT                 -- person/car/dog/cat/bird
  score           REAL
  snapshot        BLOB                 -- JPEG thumbnail for UI
  embedding       BLOB                 -- CLIP vector for this event
  match_status    TEXT                 -- "known", "unknown", "ambiguous", "unreviewed"
  matched_object_id INTEGER FK -> known_objects (nullable)
  match_confidence REAL
  ai_description  TEXT                 -- Scene description from Haiku/Ollama
  notified        BOOLEAN DEFAULT 0   -- Was a Matrix alert sent?
  created_at      TIMESTAMP

settings
  key             TEXT PRIMARY KEY
  value           TEXT                 -- JSON-encoded value
```

### Storage Estimate

~50 events/day x 50KB thumbnail = ~2.5MB/day. With 90-day retention and auto-pruning, DB stays under 500MB.

## Ollama & CLIP Integration

### clip-service (new sidecar container)

Runs on 192.168.86.196 alongside Ollama. Lightweight FastAPI app that loads the CLIP model and exposes embedding endpoints.

```
192.168.86.196 (GPU machine, RTX 3090)
  +-- Ollama (:11434) -- vision LLM (llava:13b)
  +-- clip-service (:5200) -- CLIP embedding generation
        POST /embed  {image: base64} -> {embedding: [float...]}
        POST /similarity  {embedding1, embedding2} -> {score: 0.0-1.0}
```

- Uses `open_clip` library with ViT-L/14 model
- GPU-accelerated, ~2GB VRAM, <50ms per image
- Stateless -- frigate-notify stores the embeddings

### Ollama Vision Models

Pull on 192.168.86.196:
- `llava:13b` -- scene description + ambiguous object matching
- ~7.6GB free VRAM after CLIP (3090 has 24GB total)

### Vision Backend Configuration

User-configurable in Settings page:
- **Scene description backend**: Haiku / Ollama / Ollama-primary-Haiku-fallback
- **Recognition backend**: CLIP+Ollama / CLIP+Haiku / CLIP-only

## Tech Stack

- **Python + FastAPI** -- replaces current raw HTTP server, serves API + static files
- **SQLite** -- data storage (known objects, embeddings, events, settings)
- **Vanilla HTML/CSS/JS** -- web UI, consistent with Farm Manager
- **paho-mqtt** -- MQTT listener (kept from v1)
- **aiohttp** -- Matrix client (kept from v1)
- **open_clip** -- CLIP service sidecar on GPU machine
- **numpy** -- embedding similarity calculations

## Deployment

- `frigate-notify` container: on same node as Frigate (hf-pdocker-01 when back)
- `clip-service` container: on 192.168.86.196 (GPU machine), compose file on NAS
- Ollama: already running on 192.168.86.196

### Compose Location

- frigate-notify: `/mnt/docker-data/compose/hf-pdocker-01/frigate-notify/`
- clip-service: `/mnt/docker-data/compose/gpu-machine/clip-service/`

### Farm Manager Link

Add "Frigate Alerts" entry to Farm Manager sidebar config pointing to `http://<frigate-notify-host>:5199`. No code coupling.

## Migration from v1

All existing v1 functionality preserved:
- MQTT listener and event filtering (labels, scores, dedup, rate limiting)
- Frigate snapshot fetching
- Claude Haiku vision analysis
- Matrix notifications (image + text)
- Health endpoint at `/health`

New features layer on top. The app can run in "v1 mode" (no CLIP service, no known objects) if the GPU machine is offline.

## Decisions

- **Standalone app with Farm Manager link** over built-in: independent deployment, no coupling
- **Tag from events** over upload reference photos: uses real camera footage, less setup
- **Quiet log for known objects** over full suppress: can review routine activity without being interrupted
- **Hybrid CLIP + Vision LLM** over either alone: fast matching for clear cases, LLM accuracy for ambiguous
- **User-configurable vision backend**: flexibility to use local or cloud, switch as needed
- **SQLite** over PostgreSQL: single-file, no extra container, sufficient for this data volume
- **Vanilla JS** over React/Vue: consistent with Farm Manager, no build step, simple