# LinkedStorm — Home Audio Assistant Design **Date:** 2026-05-27 **Status:** Approved --- ## Context The goal is a whole-home audio assistant that passively records household conversations, identifies who is speaking, produces searchable transcripts, and maintains a daily activity log. The motivation is to have a reliable, private, locally-hosted record of decisions and conversations that can be queried naturally ("what did we decide about the vacation?") without relying on cloud services. The system reuses as much existing homelab infrastructure as possible: Whisper STT on the GPU machine, LiveKit for WebRTC, Ollama for LLM queries, and SpeechBrain for audio ML — all already deployed. Speaker labeling mirrors the proven frigate-notify person-identification pattern (CLIP embeddings → voice embeddings, cosine similarity, three-tier matching, review-and-tag UI). --- ## Architecture **Hybrid approach:** Lightweight WebSocket audio capture from browser tabs → server-side processing pipeline → LiveKit for live monitoring and replay. ### Container Breakdown Every deployable component is its own container, following the same pattern as VoiceCoach (7 containers) and the rest of the homelab. LinkedStorm is 3 containers: | Container | Purpose | Port | Host | |---|---|---|---| | `linkedstorm-pipeline` | WebSocket ingestion, Silero VAD, Whisper calls, resemblyzer speaker embedding, cosine similarity matching, SQLite writes, LiveKit publish | 8300 | hf-pdocker-01 | | `linkedstorm-api` | FastAPI web dashboard + REST API, static UI, FTS5 search, RAG queries, speaker labeling | 8310 | hf-pdocker-01 | | `linkedstorm-cron` | Nightly Ollama summary, unknown speaker clustering, WAV clip purge | — (no port) | hf-pdocker-01 | All three share a single Docker volume for the SQLite database and clips directory. ### External Services (Reused Without Modification) | Service | What | Where | |---|---|---| | Speaches/Whisper | `POST /v1/audio/transcriptions` | GPU machine :8000 | | LiveKit | Per-room replay rooms (`linkedstorm-{room}`) | GPU machine :7880 | | Ollama | RAG queries + daily summaries (`qwen2.5:14b`) | GPU machine :11434 | Speaker voice embedding uses `resemblyzer` (ECAPA-TDNN, ~50MB) running in-process inside `linkedstorm-pipeline` — no separate microservice, no coupling to VoiceCoach. ### Data Flow ``` Browser tab (capture.html) └─ MediaRecorder → WebSocket → linkedstorm-pipeline:8300 └─ Silero VAD detects speech segment └─ [Whisper STT via Speaches ‖ resemblyzer embed] concurrent └─ cosine similarity → speaker match └─ SQLite write + WAV clip save └─ LiveKit publish → dashboard replay linkedstorm-api:8310 ← reads SQLite → serves dashboard + API linkedstorm-cron ← runs nightly at midnight via entrypoint sleep loop ``` --- ## Components ### 1. Browser Capture Page (`linkedstorm-capture`) Single HTML page opened on any device to make it a room microphone. - `MediaRecorder` at 16kHz mono, 250ms chunks streamed over WebSocket - Room name set via URL param (`?room=Kitchen`) or prompt on first load - Persistent room name stored in `localStorage` - Mute toggle: stops `MediaRecorder`, sends `{"type":"mute"}` control message, server discards buffered audio for that room - Visual indicator: green pulsing dot (streaming) / red dot (muted) / gray (disconnected) - Auto-reconnect with exponential backoff on WebSocket drop ### 2. Audio Ingestion Service (`linkedstorm-ingestion`) Python asyncio service on hf-pdocker-01. Single process, per-room coroutines. - WebSocket endpoint: `ws://linkedstorm:8300/ws/{room_id}` - Per-room `asyncio.Queue` of audio chunks - **Silero VAD** (reused from VoiceCoach): processes each 250ms chunk; accumulates on speech-detected, emits segment on ≥600ms silence; discards segments <0.5s - On segment emit: calls `process_utterance(room_id, audio_bytes, start_time, end_time)` - Room registry: upsert `rooms` table on first connection, update `last_seen` on each chunk, set `is_active=False` on disconnect ### 3. Processing Pipeline Runs as an async task per emitted speech segment. **Step 1 — Concurrent inference** (asyncio gather): - `POST http://192.168.86.150:8000/v1/audio/transcriptions` with audio WAV → transcript text - `resemblyzer.preprocess_wav()` + `encoder.embed_utterance()` in-process → voice embedding (256-dim float32 array) **Step 2 — Speaker matching:** - Load all `voice_embeddings` rows from SQLite - Compute cosine similarity between utterance embedding and each reference - Group by `speaker_id`, take MAX per speaker - Thresholds: ≥0.85 → `known`, 0.60–0.84 → `ambiguous`, <0.60 → `unknown` **Step 3 — Persist:** - Save audio clip to `/data/linkedstorm/clips/{date}/{room_id}/{utterance_id}.wav` - Insert `utterances` row: transcript, embedding, room_id, speaker_id, match_status, match_confidence, start_time, end_time, audio_clip_path **Step 4 — LiveKit publish:** - Publish audio segment to LiveKit room `linkedstorm-{room_id}` for live monitoring/replay ### 4. Web UI Service (`linkedstorm-ui`) FastAPI on hf-pdocker-01 port 8310. Static HTML/JS frontend, JSON API backend. Same pattern as frigate-notify. **API endpoints:** ``` GET /api/utterances # list with filters: date, room_id, speaker_id, match_status GET /api/utterances/{id}/clip # stream audio clip WAV POST /api/utterances/{id}/tag # label utterance: {speaker_id} or {name, create_new: true} POST /api/utterances/{id}/dismiss # mark as noise/unknown, no review needed GET /api/speakers # list all known speakers with utterance counts POST /api/speakers # create new speaker GET /api/rooms # list rooms with is_active, last_seen POST /api/rooms/{id}/mute # send mute command to room's WebSocket connection GET /api/search?q=... # full-text search across utterances.transcript (SQLite FTS5) POST /api/ask # RAG query: builds context from utterances, calls Ollama GET /api/summary/{date} # fetch or trigger daily summary for a date ``` **UI screens (5 tabs, single-page app):** 1. **Daily Log** — timeline grouped by time blocks, color-coded by speaker, inline Play/Tag/Dismiss on unknowns 2. **Search** — full-text with keyword highlighting, date-range filter 3. **Speakers** — known speaker cards + review-queue banner for unreviewed utterances 4. **Ask AI** — RAG chat backed by Ollama, responses cite room + timestamp sources 5. **Rooms** — live capture device status, per-room mute toggle, link to open capture page ### 5. Nightly Cron Jobs Run at midnight via cron on hf-pdocker-01. **Daily summary:** Fetch all utterances for the day → batch into prompt → send to Ollama (`qwen2.5:14b`) → store in `daily_summaries`. Prompt instructs the model to summarize key decisions, activities, and topics — not to transcribe verbatim. **Unknown speaker clustering:** Load all `unknown` and `ambiguous` utterance embeddings from the past 7 days → cluster by cosine similarity (threshold 0.70) → group consistent voices as `Unknown #1`, `Unknown #2`, etc. in the review queue rather than scattered individuals. --- ## Data Model SQLite at `/data/linkedstorm/linkedstorm.db`. WAL mode, foreign keys enabled. Thread-local connections (same pattern as frigate-notify). ```sql CREATE TABLE rooms ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, device_label TEXT, is_active BOOLEAN DEFAULT 0, last_seen DATETIME, created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE speakers ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, created_at DATETIME DEFAULT CURRENT_TIMESTAMP, updated_at DATETIME DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE voice_embeddings ( id INTEGER PRIMARY KEY, speaker_id INTEGER NOT NULL REFERENCES speakers(id), embedding BLOB NOT NULL, -- packed float32 audio_sample BLOB, -- short reference WAV created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); CREATE TABLE utterances ( id INTEGER PRIMARY KEY, room_id INTEGER NOT NULL REFERENCES rooms(id), speaker_id INTEGER REFERENCES speakers(id), -- nullable transcript TEXT NOT NULL, embedding BLOB, -- packed float32 match_status TEXT DEFAULT 'unreviewed', -- known/unknown/ambiguous/unreviewed match_confidence REAL, start_time DATETIME NOT NULL, end_time DATETIME NOT NULL, audio_clip_path TEXT, livekit_room TEXT, created_at DATETIME DEFAULT CURRENT_TIMESTAMP ); CREATE VIRTUAL TABLE utterances_fts USING fts5( transcript, content=utterances, content_rowid=id ); CREATE TABLE daily_summaries ( id INTEGER PRIMARY KEY, date DATE NOT NULL UNIQUE, summary_text TEXT NOT NULL, generated_at DATETIME DEFAULT CURRENT_TIMESTAMP ); ``` --- ## Speaker Identification Mirrors frigate-notify's CLIP embedding pattern, adapted for voice: | Similarity | Status | Action | |---|---|---| | ≥ 0.85 | `known` | Auto-label speaker. Store utterance embedding as additional reference. | | 0.60–0.84 | `ambiguous` | Store utterance. Appear in review queue with Play + Tag buttons. | | < 0.60 | `unknown` | Store utterance. Cluster with other unknowns nightly. | | No embed | `unreviewed` | SpeechBrain unavailable. Transcript stored, embed missing. | **Labeling flow (same as frigate-notify tag workflow):** `POST /api/utterances/{id}/tag` with `{speaker_id}` or `{name, create_new: true}` → sets `match_status='known'`, `match_confidence=1.0`, stores utterance's embedding in `voice_embeddings` for future matching. --- ## Deployment **3 containers on hf-pdocker-01** **Compose file:** `compose/hf-pdocker-01/linkedstorm/docker-compose.yml` **Ports:** - `8300` — `linkedstorm-pipeline` WebSocket ingestion (capture devices connect here) - `8310` — `linkedstorm-api` FastAPI web UI and API **Volumes:** - `/data/linkedstorm/` → `/mnt/docker-data/configs/linkedstorm/` (NAS) - `linkedstorm.db` — SQLite database - `clips/` — audio clip WAV files **Clip retention:** A VAD-triggered household produces roughly 50–200MB of WAV clips per day. The nightly cron should delete clips older than 30 days (configurable via `CLIP_RETENTION_DAYS` env var) while keeping the transcript and embedding rows permanently. The `audio_clip_path` column is set to `NULL` for purged clips; the Play button in the UI shows "clip expired" for those utterances. **Environment:** ``` SPEACHES_URL=http://192.168.86.150:8000 CLIP_RETENTION_DAYS=30 OLLAMA_URL=http://192.168.86.150:11434 LIVEKIT_URL=ws://192.168.86.150:7880 LIVEKIT_API_KEY=... LIVEKIT_API_SECRET=... ``` **Traefik:** `linkedstorm.hobbs.farm` → port 8310 (dashboard). Capture page accessed at `linkedstorm.hobbs.farm/capture?room=Kitchen`. **Cron entries on hf-pdocker-01:** ``` 0 0 * * * docker exec linkedstorm python -m cron.daily_summary 5 0 * * * docker exec linkedstorm python -m cron.cluster_unknowns 10 0 * * * docker exec linkedstorm python -m cron.purge_clips ``` --- ## File Structure ``` compose/hf-pdocker-01/linkedstorm/ ├── docker-compose.yml ├── .env.example ├── pipeline/ │ └── Dockerfile # linkedstorm-pipeline image ├── api/ │ └── Dockerfile # linkedstorm-api image └── cron/ └── Dockerfile # linkedstorm-cron image # Shared app code (mounted into pipeline + api containers, or baked into images) app/ ├── config.py # Settings from env vars (shared) ├── database.py # SQLite thread-local connections, schema init (shared) ├── models.py # Dataclasses for rooms, speakers, utterances (shared) │ ├── pipeline/ # linkedstorm-pipeline │ ├── main.py # asyncio entrypoint: WebSocket server + per-room VAD loops │ ├── ingestion.py # WebSocket handler, per-room audio buffer + Silero VAD │ ├── processor.py # process_utterance: Whisper + resemblyzer + matching + persist │ ├── recognition.py # resemblyzer embed + cosine similarity, speaker matching │ └── livekit.py # Publish audio segments to LiveKit rooms │ ├── api/ # linkedstorm-api │ ├── main.py # FastAPI entrypoint, mounts routes, serves static files │ ├── routes/ │ │ ├── utterances.py # /api/utterances CRUD + tag + dismiss │ │ ├── speakers.py # /api/speakers CRUD │ │ ├── rooms.py # /api/rooms status + mute relay to pipeline │ │ ├── search.py # /api/search (SQLite FTS5) │ │ └── rag.py # /api/ask, /api/summary │ └── static/ │ ├── index.html # Dashboard SPA (5-tab layout) │ ├── capture.html # Minimal capture page (one per room device) │ └── app.js / style.css │ └── cron/ # linkedstorm-cron ├── main.py # Sleep-loop entrypoint: runs all jobs at midnight ├── daily_summary.py # Ollama daily summary generation ├── cluster_unknowns.py # Nightly unknown speaker clustering └── purge_clips.py # Delete WAV files older than CLIP_RETENTION_DAYS ``` --- ## Verification 1. **Capture → ingestion:** Open `capture.html?room=Kitchen` in a browser, speak — confirm WebSocket connects, room appears active in the Rooms tab of the dashboard 2. **VAD:** Speak briefly then stay silent — confirm a single utterance appears in the log (not multiple fragments), and background noise is not logged 3. **Transcription:** Read a known sentence — confirm transcript text in the utterance matches 4. **Speaker labeling:** Tag an unknown utterance as "Jeremy" → speak again → confirm next utterances auto-resolve to Jeremy with high confidence 5. **Mute:** Click mute on the capture page — confirm no utterances logged while muted, muted status shown in Rooms tab 6. **Search:** Search for a word spoken earlier — confirm it appears highlighted in results 7. **Ask AI:** Ask "What did we talk about today?" — confirm Ollama returns a coherent summary with room/time citations 8. **LiveKit replay:** From the dashboard, join a room's LiveKit channel — confirm processed audio plays back 9. **Nightly cron:** Trigger `daily_summary` manually — confirm `daily_summaries` row appears and is shown in the log view 10. **Prometheus probe:** Add `http://192.168.86.192:8310/health` to blackbox targets and confirm it's green