infra: full IaC state after hf-pdocker-01 rebuild (2026-03-08)
Replaces initial Nomad/Consul/Terraform skeleton with current Docker-based infrastructure across all three nodes (hf-pdocker-01, hf-pdocker-02, bart). Key changes: - CLAUDE.md: updated service distribution reflecting hf-pdocker-01 restored - docs/plans/2026-03-08-hf-pdocker-01-rebuild.md: new rebuild plan doc - compose/bart/: ai-agents, guacamole, matrix, media, openvas, smarthome, watchtower - compose/hf-pdocker-02/: traefik, monitoring, adguard, gitea, semaphore, wikijs, unifi, vaultwarden, actual-budget, mcpo, netclaw, trivy, watchtower, n8n - compose/hf-pdocker-01/: frigate, frigate-notify, farm-agent - stacks/: all Swarm stack definitions (monitoring, traefik, smarthome, matrix, guacamole, media, ai-agents, frigate, gitea, semaphore, wikijs, unifi, etc.) - .gitignore: exclude .env credential files and acme.json certs - .env.example: placeholder credential files for all services - .claude/agents/: gitea-backup and other infrastructure agent configs - scripts/: frigate media cleanup and NAS sync scripts Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,242 @@
|
||||
# Frigate-Notify v2 Design
|
||||
|
||||
## Summary
|
||||
|
||||
Evolve frigate-notify from a simple MQTT-to-Matrix alert forwarder into an intelligent notification system with known-object recognition, a web UI for managing identities and reviewing events, and configurable vision backends (local Ollama on a 3090 GPU or Claude Haiku cloud API).
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Reduce noise** — stop alerting on known family members, vehicles, and pets
|
||||
2. **Learn from real events** — tag objects from Frigate detection snapshots to build a recognition database over time
|
||||
3. **Local AI processing** — use a local 3090 GPU (192.168.86.196) for CLIP embeddings and vision LLM, reducing cloud API costs
|
||||
4. **Web interface** — standalone app for managing known objects, reviewing events, and configuring settings
|
||||
5. **Farm Manager integration** — sidebar link in Farm Manager pointing to the frigate-notify UI
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Frigate NVR ──MQTT──> Mosquitto Broker
|
||||
|
|
||||
v
|
||||
frigate-notify v2 (:5199)
|
||||
+-------------------------+
|
||||
| MQTT Listener |
|
||||
| Event Processor |
|
||||
| Web UI (HTML/CSS/JS) |
|
||||
| REST API (FastAPI) |
|
||||
| SQLite DB |
|
||||
+------+------+------+-----+
|
||||
| | |
|
||||
+------------+ | +------------+
|
||||
v v v
|
||||
clip-service (:5200) Ollama (:11434) Claude Haiku API
|
||||
192.168.86.196 192.168.86.196 cloud
|
||||
CLIP embeddings Vision LLM Vision fallback
|
||||
Scene descriptions Scene descriptions
|
||||
| |
|
||||
+-------+-----------+
|
||||
v
|
||||
GPU Machine (RTX 3090, 24GB VRAM)
|
||||
|
||||
frigate-notify v2
|
||||
|
|
||||
v
|
||||
Matrix API
|
||||
#video-alerts room
|
||||
(unknown objects only)
|
||||
|
||||
Farm Manager (:8888) ──sidebar link──> frigate-notify (:5199)
|
||||
```
|
||||
|
||||
## Known Object Recognition Pipeline
|
||||
|
||||
Each Frigate event goes through this flow:
|
||||
|
||||
```
|
||||
Event arrives (MQTT, type="end")
|
||||
-> Fetch snapshot from Frigate API
|
||||
-> Generate CLIP embedding via clip-service (192.168.86.196:5200)
|
||||
-> Compare against stored embeddings in SQLite
|
||||
-> High confidence match (>0.85): "known" -- log quietly, no alert
|
||||
-> Low confidence (<0.60): "unknown" -- send Matrix alert
|
||||
-> Ambiguous (0.60-0.85): send snapshot + reference images to Vision LLM
|
||||
-> LLM confirms match: "known" -- log quietly
|
||||
-> LLM says no match: "unknown" -- send Matrix alert
|
||||
```
|
||||
|
||||
Scene description (the AI-generated text in notifications) is handled separately and is configurable to use either Haiku or Ollama vision model.
|
||||
|
||||
### Fallback Chain
|
||||
|
||||
1. clip-service down: skip embedding matching, send all events to vision LLM
|
||||
2. Ollama down: fall back to Claude Haiku for everything
|
||||
3. Haiku down: structured text description ("Person detected on back_porch")
|
||||
|
||||
### Known Object Types
|
||||
|
||||
- **People** -- tagged by name, matched by face/body appearance
|
||||
- **Vehicles** -- tagged by description (e.g., "blue F-150"), matched by shape/color
|
||||
- **Pets** -- tagged by name, matched by appearance
|
||||
|
||||
### Learning Flow
|
||||
|
||||
1. Events stream in and show up in the web UI's "Recent Events" feed
|
||||
2. User taps an event and chooses "Tag as known" -> enters name/description -> selects type (person/vehicle/pet)
|
||||
3. System stores the CLIP embedding from that snapshot tied to that identity
|
||||
4. Multiple tagged snapshots of the same identity improve matching accuracy over time
|
||||
5. User can also "Confirm" or "Reject" the system's auto-matches to refine it
|
||||
|
||||
### Notification Behavior for Known Objects
|
||||
|
||||
- **Known objects**: logged quietly in the web UI, no Matrix notification
|
||||
- **Unknown objects**: full Matrix alert with snapshot + AI description (existing behavior)
|
||||
|
||||
## Web UI
|
||||
|
||||
Served at `:5199`, vanilla HTML/CSS/JS (consistent with Farm Manager). Same container as the API and event processor.
|
||||
|
||||
### Pages
|
||||
|
||||
| Page | Purpose |
|
||||
|------|---------|
|
||||
| Dashboard | Live feed of recent events. Each shows snapshot thumbnail, camera, label, score, known/unknown status, AI description. Filter by camera, label, status. |
|
||||
| Known Objects | Grid of all tagged identities. Click to see all associated snapshots, edit name, delete. Organized by type (people/vehicles/pets). |
|
||||
| Review Queue | Unreviewed events needing tagging. Quick-action buttons: "Tag as known", "Dismiss", "Confirm match". |
|
||||
| Settings | Vision backend toggle (Haiku/Ollama/both), confidence thresholds, rate limiting, camera enable/disable, Matrix notification preferences. |
|
||||
|
||||
## REST API
|
||||
|
||||
| Endpoint | Method | Purpose |
|
||||
|----------|--------|---------|
|
||||
| `/api/events` | GET | List recent events, filterable |
|
||||
| `/api/events/{id}` | GET | Single event detail |
|
||||
| `/api/events/{id}/tag` | POST | Tag event as known object |
|
||||
| `/api/events/{id}/dismiss` | POST | Dismiss event |
|
||||
| `/api/known-objects` | GET | List all known identities |
|
||||
| `/api/known-objects/{id}` | GET/PUT/DELETE | Manage a known identity |
|
||||
| `/api/settings` | GET/PUT | Read/update configuration |
|
||||
| `/api/stats` | GET | Event counts, match rates |
|
||||
| `/health` | GET | Existing health endpoint (preserved) |
|
||||
|
||||
## Data Model
|
||||
|
||||
SQLite database at `/mnt/docker-data/configs/frigate-notify/frigate-notify.db`
|
||||
|
||||
### Tables
|
||||
|
||||
```sql
|
||||
known_objects
|
||||
id INTEGER PRIMARY KEY
|
||||
name TEXT NOT NULL -- "Jeremy", "Blue F-150", "Max the dog"
|
||||
type TEXT NOT NULL -- "person", "vehicle", "pet"
|
||||
action TEXT DEFAULT "quiet" -- "quiet" (log only) | "suppress" | "alert"
|
||||
created_at TIMESTAMP
|
||||
updated_at TIMESTAMP
|
||||
|
||||
embeddings
|
||||
id INTEGER PRIMARY KEY
|
||||
known_object_id INTEGER FK -> known_objects
|
||||
embedding BLOB -- CLIP vector (float32 array)
|
||||
snapshot_url TEXT -- Frigate snapshot URL for reference
|
||||
thumbnail BLOB -- Small JPEG for UI display
|
||||
created_at TIMESTAMP
|
||||
|
||||
events
|
||||
id INTEGER PRIMARY KEY
|
||||
frigate_event_id TEXT UNIQUE -- Frigate's event ID
|
||||
camera TEXT
|
||||
label TEXT -- person/car/dog/cat/bird
|
||||
score REAL
|
||||
snapshot BLOB -- JPEG thumbnail for UI
|
||||
embedding BLOB -- CLIP vector for this event
|
||||
match_status TEXT -- "known", "unknown", "ambiguous", "unreviewed"
|
||||
matched_object_id INTEGER FK -> known_objects (nullable)
|
||||
match_confidence REAL
|
||||
ai_description TEXT -- Scene description from Haiku/Ollama
|
||||
notified BOOLEAN DEFAULT 0 -- Was a Matrix alert sent?
|
||||
created_at TIMESTAMP
|
||||
|
||||
settings
|
||||
key TEXT PRIMARY KEY
|
||||
value TEXT -- JSON-encoded value
|
||||
```
|
||||
|
||||
### Storage Estimate
|
||||
|
||||
~50 events/day x 50KB thumbnail = ~2.5MB/day. With 90-day retention and auto-pruning, DB stays under 500MB.
|
||||
|
||||
## Ollama & CLIP Integration
|
||||
|
||||
### clip-service (new sidecar container)
|
||||
|
||||
Runs on 192.168.86.196 alongside Ollama. Lightweight FastAPI app that loads the CLIP model and exposes embedding endpoints.
|
||||
|
||||
```
|
||||
192.168.86.196 (GPU machine, RTX 3090)
|
||||
+-- Ollama (:11434) -- vision LLM (llava:13b)
|
||||
+-- clip-service (:5200) -- CLIP embedding generation
|
||||
POST /embed {image: base64} -> {embedding: [float...]}
|
||||
POST /similarity {embedding1, embedding2} -> {score: 0.0-1.0}
|
||||
```
|
||||
|
||||
- Uses `open_clip` library with ViT-L/14 model
|
||||
- GPU-accelerated, ~2GB VRAM, <50ms per image
|
||||
- Stateless -- frigate-notify stores the embeddings
|
||||
|
||||
### Ollama Vision Models
|
||||
|
||||
Pull on 192.168.86.196:
|
||||
- `llava:13b` -- scene description + ambiguous object matching
|
||||
- ~7.6GB free VRAM after CLIP (3090 has 24GB total)
|
||||
|
||||
### Vision Backend Configuration
|
||||
|
||||
User-configurable in Settings page:
|
||||
- **Scene description backend**: Haiku / Ollama / Ollama-primary-Haiku-fallback
|
||||
- **Recognition backend**: CLIP+Ollama / CLIP+Haiku / CLIP-only
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Python + FastAPI** -- replaces current raw HTTP server, serves API + static files
|
||||
- **SQLite** -- data storage (known objects, embeddings, events, settings)
|
||||
- **Vanilla HTML/CSS/JS** -- web UI, consistent with Farm Manager
|
||||
- **paho-mqtt** -- MQTT listener (kept from v1)
|
||||
- **aiohttp** -- Matrix client (kept from v1)
|
||||
- **open_clip** -- CLIP service sidecar on GPU machine
|
||||
- **numpy** -- embedding similarity calculations
|
||||
|
||||
## Deployment
|
||||
|
||||
- `frigate-notify` container: on same node as Frigate (hf-pdocker-01 when back)
|
||||
- `clip-service` container: on 192.168.86.196 (GPU machine), compose file on NAS
|
||||
- Ollama: already running on 192.168.86.196
|
||||
|
||||
### Compose Location
|
||||
|
||||
- frigate-notify: `/mnt/docker-data/compose/hf-pdocker-01/frigate-notify/`
|
||||
- clip-service: `/mnt/docker-data/compose/gpu-machine/clip-service/`
|
||||
|
||||
### Farm Manager Link
|
||||
|
||||
Add "Frigate Alerts" entry to Farm Manager sidebar config pointing to `http://<frigate-notify-host>:5199`. No code coupling.
|
||||
|
||||
## Migration from v1
|
||||
|
||||
All existing v1 functionality preserved:
|
||||
- MQTT listener and event filtering (labels, scores, dedup, rate limiting)
|
||||
- Frigate snapshot fetching
|
||||
- Claude Haiku vision analysis
|
||||
- Matrix notifications (image + text)
|
||||
- Health endpoint at `/health`
|
||||
|
||||
New features layer on top. The app can run in "v1 mode" (no CLIP service, no known objects) if the GPU machine is offline.
|
||||
|
||||
## Decisions
|
||||
|
||||
- **Standalone app with Farm Manager link** over built-in: independent deployment, no coupling
|
||||
- **Tag from events** over upload reference photos: uses real camera footage, less setup
|
||||
- **Quiet log for known objects** over full suppress: can review routine activity without being interrupted
|
||||
- **Hybrid CLIP + Vision LLM** over either alone: fast matching for clear cases, LLM accuracy for ambiguous
|
||||
- **User-configurable vision backend**: flexibility to use local or cloud, switch as needed
|
||||
- **SQLite** over PostgreSQL: single-file, no extra container, sufficient for this data volume
|
||||
- **Vanilla JS** over React/Vue: consistent with Farm Manager, no build step, simple
|
||||
Reference in New Issue
Block a user