From cd3f3161da625bb577d6ce9e12a0f83bcffa8d8f Mon Sep 17 00:00:00 2001 From: jeremy Date: Wed, 27 May 2026 21:59:32 -0500 Subject: [PATCH] docs: add LinkedStorm home audio assistant design spec Co-Authored-By: Claude Sonnet 4.6 --- .../specs/2026-05-27-linkedstorm-design.md | 303 ++++++++++++++++++ 1 file changed, 303 insertions(+) create mode 100644 docs/superpowers/specs/2026-05-27-linkedstorm-design.md diff --git a/docs/superpowers/specs/2026-05-27-linkedstorm-design.md b/docs/superpowers/specs/2026-05-27-linkedstorm-design.md new file mode 100644 index 0000000..5aa8d84 --- /dev/null +++ b/docs/superpowers/specs/2026-05-27-linkedstorm-design.md @@ -0,0 +1,303 @@ +# LinkedStorm — Home Audio Assistant Design + +**Date:** 2026-05-27 +**Status:** Approved + +--- + +## Context + +The goal is a whole-home audio assistant that passively records household conversations, identifies who is speaking, produces searchable transcripts, and maintains a daily activity log. The motivation is to have a reliable, private, locally-hosted record of decisions and conversations that can be queried naturally ("what did we decide about the vacation?") without relying on cloud services. + +The system reuses as much existing homelab infrastructure as possible: Whisper STT on the GPU machine, LiveKit for WebRTC, Ollama for LLM queries, and SpeechBrain for audio ML — all already deployed. Speaker labeling mirrors the proven frigate-notify person-identification pattern (CLIP embeddings → voice embeddings, cosine similarity, three-tier matching, review-and-tag UI). + +--- + +## Architecture + +**Hybrid approach:** Lightweight WebSocket audio capture from browser tabs → server-side processing pipeline → LiveKit for live monitoring and replay. + +### Container Breakdown + +Every deployable component is its own container, following the same pattern as VoiceCoach (7 containers) and the rest of the homelab. LinkedStorm is 3 containers: + +| Container | Purpose | Port | Host | +|---|---|---|---| +| `linkedstorm-pipeline` | WebSocket ingestion, Silero VAD, Whisper calls, resemblyzer speaker embedding, cosine similarity matching, SQLite writes, LiveKit publish | 8300 | hf-pdocker-01 | +| `linkedstorm-api` | FastAPI web dashboard + REST API, static UI, FTS5 search, RAG queries, speaker labeling | 8310 | hf-pdocker-01 | +| `linkedstorm-cron` | Nightly Ollama summary, unknown speaker clustering, WAV clip purge | — (no port) | hf-pdocker-01 | + +All three share a single Docker volume for the SQLite database and clips directory. + +### External Services (Reused Without Modification) + +| Service | What | Where | +|---|---|---| +| Speaches/Whisper | `POST /v1/audio/transcriptions` | GPU machine :8000 | +| LiveKit | Per-room replay rooms (`linkedstorm-{room}`) | GPU machine :7880 | +| Ollama | RAG queries + daily summaries (`qwen2.5:14b`) | GPU machine :11434 | + +Speaker voice embedding uses `resemblyzer` (ECAPA-TDNN, ~50MB) running in-process inside `linkedstorm-pipeline` — no separate microservice, no coupling to VoiceCoach. + +### Data Flow + +``` +Browser tab (capture.html) + └─ MediaRecorder → WebSocket → linkedstorm-pipeline:8300 + └─ Silero VAD detects speech segment + └─ [Whisper STT via Speaches ‖ resemblyzer embed] concurrent + └─ cosine similarity → speaker match + └─ SQLite write + WAV clip save + └─ LiveKit publish → dashboard replay +linkedstorm-api:8310 ← reads SQLite → serves dashboard + API +linkedstorm-cron ← runs nightly at midnight via entrypoint sleep loop +``` + +--- + +## Components + +### 1. Browser Capture Page (`linkedstorm-capture`) +Single HTML page opened on any device to make it a room microphone. + +- `MediaRecorder` at 16kHz mono, 250ms chunks streamed over WebSocket +- Room name set via URL param (`?room=Kitchen`) or prompt on first load +- Persistent room name stored in `localStorage` +- Mute toggle: stops `MediaRecorder`, sends `{"type":"mute"}` control message, server discards buffered audio for that room +- Visual indicator: green pulsing dot (streaming) / red dot (muted) / gray (disconnected) +- Auto-reconnect with exponential backoff on WebSocket drop + +### 2. Audio Ingestion Service (`linkedstorm-ingestion`) +Python asyncio service on hf-pdocker-01. Single process, per-room coroutines. + +- WebSocket endpoint: `ws://linkedstorm:8300/ws/{room_id}` +- Per-room `asyncio.Queue` of audio chunks +- **Silero VAD** (reused from VoiceCoach): processes each 250ms chunk; accumulates on speech-detected, emits segment on ≥600ms silence; discards segments <0.5s +- On segment emit: calls `process_utterance(room_id, audio_bytes, start_time, end_time)` +- Room registry: upsert `rooms` table on first connection, update `last_seen` on each chunk, set `is_active=False` on disconnect + +### 3. Processing Pipeline +Runs as an async task per emitted speech segment. + +**Step 1 — Concurrent inference** (asyncio gather): +- `POST http://192.168.86.150:8000/v1/audio/transcriptions` with audio WAV → transcript text +- `resemblyzer.preprocess_wav()` + `encoder.embed_utterance()` in-process → voice embedding (256-dim float32 array) + +**Step 2 — Speaker matching:** +- Load all `voice_embeddings` rows from SQLite +- Compute cosine similarity between utterance embedding and each reference +- Group by `speaker_id`, take MAX per speaker +- Thresholds: ≥0.85 → `known`, 0.60–0.84 → `ambiguous`, <0.60 → `unknown` + +**Step 3 — Persist:** +- Save audio clip to `/data/linkedstorm/clips/{date}/{room_id}/{utterance_id}.wav` +- Insert `utterances` row: transcript, embedding, room_id, speaker_id, match_status, match_confidence, start_time, end_time, audio_clip_path + +**Step 4 — LiveKit publish:** +- Publish audio segment to LiveKit room `linkedstorm-{room_id}` for live monitoring/replay + +### 4. Web UI Service (`linkedstorm-ui`) +FastAPI on hf-pdocker-01 port 8310. Static HTML/JS frontend, JSON API backend. Same pattern as frigate-notify. + +**API endpoints:** +``` +GET /api/utterances # list with filters: date, room_id, speaker_id, match_status +GET /api/utterances/{id}/clip # stream audio clip WAV +POST /api/utterances/{id}/tag # label utterance: {speaker_id} or {name, create_new: true} +POST /api/utterances/{id}/dismiss # mark as noise/unknown, no review needed +GET /api/speakers # list all known speakers with utterance counts +POST /api/speakers # create new speaker +GET /api/rooms # list rooms with is_active, last_seen +POST /api/rooms/{id}/mute # send mute command to room's WebSocket connection +GET /api/search?q=... # full-text search across utterances.transcript (SQLite FTS5) +POST /api/ask # RAG query: builds context from utterances, calls Ollama +GET /api/summary/{date} # fetch or trigger daily summary for a date +``` + +**UI screens (5 tabs, single-page app):** +1. **Daily Log** — timeline grouped by time blocks, color-coded by speaker, inline Play/Tag/Dismiss on unknowns +2. **Search** — full-text with keyword highlighting, date-range filter +3. **Speakers** — known speaker cards + review-queue banner for unreviewed utterances +4. **Ask AI** — RAG chat backed by Ollama, responses cite room + timestamp sources +5. **Rooms** — live capture device status, per-room mute toggle, link to open capture page + +### 5. Nightly Cron Jobs +Run at midnight via cron on hf-pdocker-01. + +**Daily summary:** Fetch all utterances for the day → batch into prompt → send to Ollama (`qwen2.5:14b`) → store in `daily_summaries`. Prompt instructs the model to summarize key decisions, activities, and topics — not to transcribe verbatim. + +**Unknown speaker clustering:** Load all `unknown` and `ambiguous` utterance embeddings from the past 7 days → cluster by cosine similarity (threshold 0.70) → group consistent voices as `Unknown #1`, `Unknown #2`, etc. in the review queue rather than scattered individuals. + +--- + +## Data Model + +SQLite at `/data/linkedstorm/linkedstorm.db`. WAL mode, foreign keys enabled. Thread-local connections (same pattern as frigate-notify). + +```sql +CREATE TABLE rooms ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + device_label TEXT, + is_active BOOLEAN DEFAULT 0, + last_seen DATETIME, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE speakers ( + id INTEGER PRIMARY KEY, + name TEXT NOT NULL, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP, + updated_at DATETIME DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE voice_embeddings ( + id INTEGER PRIMARY KEY, + speaker_id INTEGER NOT NULL REFERENCES speakers(id), + embedding BLOB NOT NULL, -- packed float32 + audio_sample BLOB, -- short reference WAV + created_at DATETIME DEFAULT CURRENT_TIMESTAMP +); + +CREATE TABLE utterances ( + id INTEGER PRIMARY KEY, + room_id INTEGER NOT NULL REFERENCES rooms(id), + speaker_id INTEGER REFERENCES speakers(id), -- nullable + transcript TEXT NOT NULL, + embedding BLOB, -- packed float32 + match_status TEXT DEFAULT 'unreviewed', -- known/unknown/ambiguous/unreviewed + match_confidence REAL, + start_time DATETIME NOT NULL, + end_time DATETIME NOT NULL, + audio_clip_path TEXT, + livekit_room TEXT, + created_at DATETIME DEFAULT CURRENT_TIMESTAMP +); + +CREATE VIRTUAL TABLE utterances_fts USING fts5( + transcript, content=utterances, content_rowid=id +); + +CREATE TABLE daily_summaries ( + id INTEGER PRIMARY KEY, + date DATE NOT NULL UNIQUE, + summary_text TEXT NOT NULL, + generated_at DATETIME DEFAULT CURRENT_TIMESTAMP +); +``` + +--- + +## Speaker Identification + +Mirrors frigate-notify's CLIP embedding pattern, adapted for voice: + +| Similarity | Status | Action | +|---|---|---| +| ≥ 0.85 | `known` | Auto-label speaker. Store utterance embedding as additional reference. | +| 0.60–0.84 | `ambiguous` | Store utterance. Appear in review queue with Play + Tag buttons. | +| < 0.60 | `unknown` | Store utterance. Cluster with other unknowns nightly. | +| No embed | `unreviewed` | SpeechBrain unavailable. Transcript stored, embed missing. | + +**Labeling flow (same as frigate-notify tag workflow):** +`POST /api/utterances/{id}/tag` with `{speaker_id}` or `{name, create_new: true}` → +sets `match_status='known'`, `match_confidence=1.0`, stores utterance's embedding in `voice_embeddings` for future matching. + +--- + +## Deployment + +**3 containers on hf-pdocker-01** +**Compose file:** `compose/hf-pdocker-01/linkedstorm/docker-compose.yml` +**Ports:** +- `8300` — `linkedstorm-pipeline` WebSocket ingestion (capture devices connect here) +- `8310` — `linkedstorm-api` FastAPI web UI and API + +**Volumes:** +- `/data/linkedstorm/` → `/mnt/docker-data/configs/linkedstorm/` (NAS) + - `linkedstorm.db` — SQLite database + - `clips/` — audio clip WAV files + +**Clip retention:** A VAD-triggered household produces roughly 50–200MB of WAV clips per day. The nightly cron should delete clips older than 30 days (configurable via `CLIP_RETENTION_DAYS` env var) while keeping the transcript and embedding rows permanently. The `audio_clip_path` column is set to `NULL` for purged clips; the Play button in the UI shows "clip expired" for those utterances. + +**Environment:** +``` +SPEACHES_URL=http://192.168.86.150:8000 +CLIP_RETENTION_DAYS=30 +OLLAMA_URL=http://192.168.86.150:11434 +LIVEKIT_URL=ws://192.168.86.150:7880 +LIVEKIT_API_KEY=... +LIVEKIT_API_SECRET=... +``` + +**Traefik:** `linkedstorm.hobbs.farm` → port 8310 (dashboard). Capture page accessed at `linkedstorm.hobbs.farm/capture?room=Kitchen`. + +**Cron entries on hf-pdocker-01:** +``` +0 0 * * * docker exec linkedstorm python -m cron.daily_summary +5 0 * * * docker exec linkedstorm python -m cron.cluster_unknowns +10 0 * * * docker exec linkedstorm python -m cron.purge_clips +``` + +--- + +## File Structure + +``` +compose/hf-pdocker-01/linkedstorm/ +├── docker-compose.yml +├── .env.example +├── pipeline/ +│ └── Dockerfile # linkedstorm-pipeline image +├── api/ +│ └── Dockerfile # linkedstorm-api image +└── cron/ + └── Dockerfile # linkedstorm-cron image + +# Shared app code (mounted into pipeline + api containers, or baked into images) +app/ +├── config.py # Settings from env vars (shared) +├── database.py # SQLite thread-local connections, schema init (shared) +├── models.py # Dataclasses for rooms, speakers, utterances (shared) +│ +├── pipeline/ # linkedstorm-pipeline +│ ├── main.py # asyncio entrypoint: WebSocket server + per-room VAD loops +│ ├── ingestion.py # WebSocket handler, per-room audio buffer + Silero VAD +│ ├── processor.py # process_utterance: Whisper + resemblyzer + matching + persist +│ ├── recognition.py # resemblyzer embed + cosine similarity, speaker matching +│ └── livekit.py # Publish audio segments to LiveKit rooms +│ +├── api/ # linkedstorm-api +│ ├── main.py # FastAPI entrypoint, mounts routes, serves static files +│ ├── routes/ +│ │ ├── utterances.py # /api/utterances CRUD + tag + dismiss +│ │ ├── speakers.py # /api/speakers CRUD +│ │ ├── rooms.py # /api/rooms status + mute relay to pipeline +│ │ ├── search.py # /api/search (SQLite FTS5) +│ │ └── rag.py # /api/ask, /api/summary +│ └── static/ +│ ├── index.html # Dashboard SPA (5-tab layout) +│ ├── capture.html # Minimal capture page (one per room device) +│ └── app.js / style.css +│ +└── cron/ # linkedstorm-cron + ├── main.py # Sleep-loop entrypoint: runs all jobs at midnight + ├── daily_summary.py # Ollama daily summary generation + ├── cluster_unknowns.py # Nightly unknown speaker clustering + └── purge_clips.py # Delete WAV files older than CLIP_RETENTION_DAYS +``` + +--- + +## Verification + +1. **Capture → ingestion:** Open `capture.html?room=Kitchen` in a browser, speak — confirm WebSocket connects, room appears active in the Rooms tab of the dashboard +2. **VAD:** Speak briefly then stay silent — confirm a single utterance appears in the log (not multiple fragments), and background noise is not logged +3. **Transcription:** Read a known sentence — confirm transcript text in the utterance matches +4. **Speaker labeling:** Tag an unknown utterance as "Jeremy" → speak again → confirm next utterances auto-resolve to Jeremy with high confidence +5. **Mute:** Click mute on the capture page — confirm no utterances logged while muted, muted status shown in Rooms tab +6. **Search:** Search for a word spoken earlier — confirm it appears highlighted in results +7. **Ask AI:** Ask "What did we talk about today?" — confirm Ollama returns a coherent summary with room/time citations +8. **LiveKit replay:** From the dashboard, join a room's LiveKit channel — confirm processed audio plays back +9. **Nightly cron:** Trigger `daily_summary` manually — confirm `daily_summaries` row appears and is shown in the log view +10. **Prometheus probe:** Add `http://192.168.86.192:8310/health` to blackbox targets and confirm it's green