304 lines
14 KiB
Markdown
304 lines
14 KiB
Markdown
# LinkedStorm — Home Audio Assistant Design
|
||
|
||
**Date:** 2026-05-27
|
||
**Status:** Approved
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
The goal is a whole-home audio assistant that passively records household conversations, identifies who is speaking, produces searchable transcripts, and maintains a daily activity log. The motivation is to have a reliable, private, locally-hosted record of decisions and conversations that can be queried naturally ("what did we decide about the vacation?") without relying on cloud services.
|
||
|
||
The system reuses as much existing homelab infrastructure as possible: Whisper STT on the GPU machine, LiveKit for WebRTC, Ollama for LLM queries, and SpeechBrain for audio ML — all already deployed. Speaker labeling mirrors the proven frigate-notify person-identification pattern (CLIP embeddings → voice embeddings, cosine similarity, three-tier matching, review-and-tag UI).
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
**Hybrid approach:** Lightweight WebSocket audio capture from browser tabs → server-side processing pipeline → LiveKit for live monitoring and replay.
|
||
|
||
### Container Breakdown
|
||
|
||
Every deployable component is its own container, following the same pattern as VoiceCoach (7 containers) and the rest of the homelab. LinkedStorm is 3 containers:
|
||
|
||
| Container | Purpose | Port | Host |
|
||
|---|---|---|---|
|
||
| `linkedstorm-pipeline` | WebSocket ingestion, Silero VAD, Whisper calls, resemblyzer speaker embedding, cosine similarity matching, SQLite writes, LiveKit publish | 8300 | hf-pdocker-01 |
|
||
| `linkedstorm-api` | FastAPI web dashboard + REST API, static UI, FTS5 search, RAG queries, speaker labeling | 8310 | hf-pdocker-01 |
|
||
| `linkedstorm-cron` | Nightly Ollama summary, unknown speaker clustering, WAV clip purge | — (no port) | hf-pdocker-01 |
|
||
|
||
All three share a single Docker volume for the SQLite database and clips directory.
|
||
|
||
### External Services (Reused Without Modification)
|
||
|
||
| Service | What | Where |
|
||
|---|---|---|
|
||
| Speaches/Whisper | `POST /v1/audio/transcriptions` | GPU machine :8000 |
|
||
| LiveKit | Per-room replay rooms (`linkedstorm-{room}`) | GPU machine :7880 |
|
||
| Ollama | RAG queries + daily summaries (`qwen2.5:14b`) | GPU machine :11434 |
|
||
|
||
Speaker voice embedding uses `resemblyzer` (ECAPA-TDNN, ~50MB) running in-process inside `linkedstorm-pipeline` — no separate microservice, no coupling to VoiceCoach.
|
||
|
||
### Data Flow
|
||
|
||
```
|
||
Browser tab (capture.html)
|
||
└─ MediaRecorder → WebSocket → linkedstorm-pipeline:8300
|
||
└─ Silero VAD detects speech segment
|
||
└─ [Whisper STT via Speaches ‖ resemblyzer embed] concurrent
|
||
└─ cosine similarity → speaker match
|
||
└─ SQLite write + WAV clip save
|
||
└─ LiveKit publish → dashboard replay
|
||
linkedstorm-api:8310 ← reads SQLite → serves dashboard + API
|
||
linkedstorm-cron ← runs nightly at midnight via entrypoint sleep loop
|
||
```
|
||
|
||
---
|
||
|
||
## Components
|
||
|
||
### 1. Browser Capture Page (`linkedstorm-capture`)
|
||
Single HTML page opened on any device to make it a room microphone.
|
||
|
||
- `MediaRecorder` at 16kHz mono, 250ms chunks streamed over WebSocket
|
||
- Room name set via URL param (`?room=Kitchen`) or prompt on first load
|
||
- Persistent room name stored in `localStorage`
|
||
- Mute toggle: stops `MediaRecorder`, sends `{"type":"mute"}` control message, server discards buffered audio for that room
|
||
- Visual indicator: green pulsing dot (streaming) / red dot (muted) / gray (disconnected)
|
||
- Auto-reconnect with exponential backoff on WebSocket drop
|
||
|
||
### 2. Audio Ingestion Service (`linkedstorm-ingestion`)
|
||
Python asyncio service on hf-pdocker-01. Single process, per-room coroutines.
|
||
|
||
- WebSocket endpoint: `ws://linkedstorm:8300/ws/{room_id}`
|
||
- Per-room `asyncio.Queue` of audio chunks
|
||
- **Silero VAD** (reused from VoiceCoach): processes each 250ms chunk; accumulates on speech-detected, emits segment on ≥600ms silence; discards segments <0.5s
|
||
- On segment emit: calls `process_utterance(room_id, audio_bytes, start_time, end_time)`
|
||
- Room registry: upsert `rooms` table on first connection, update `last_seen` on each chunk, set `is_active=False` on disconnect
|
||
|
||
### 3. Processing Pipeline
|
||
Runs as an async task per emitted speech segment.
|
||
|
||
**Step 1 — Concurrent inference** (asyncio gather):
|
||
- `POST http://192.168.86.150:8000/v1/audio/transcriptions` with audio WAV → transcript text
|
||
- `resemblyzer.preprocess_wav()` + `encoder.embed_utterance()` in-process → voice embedding (256-dim float32 array)
|
||
|
||
**Step 2 — Speaker matching:**
|
||
- Load all `voice_embeddings` rows from SQLite
|
||
- Compute cosine similarity between utterance embedding and each reference
|
||
- Group by `speaker_id`, take MAX per speaker
|
||
- Thresholds: ≥0.85 → `known`, 0.60–0.84 → `ambiguous`, <0.60 → `unknown`
|
||
|
||
**Step 3 — Persist:**
|
||
- Save audio clip to `/data/linkedstorm/clips/{date}/{room_id}/{utterance_id}.wav`
|
||
- Insert `utterances` row: transcript, embedding, room_id, speaker_id, match_status, match_confidence, start_time, end_time, audio_clip_path
|
||
|
||
**Step 4 — LiveKit publish:**
|
||
- Publish audio segment to LiveKit room `linkedstorm-{room_id}` for live monitoring/replay
|
||
|
||
### 4. Web UI Service (`linkedstorm-ui`)
|
||
FastAPI on hf-pdocker-01 port 8310. Static HTML/JS frontend, JSON API backend. Same pattern as frigate-notify.
|
||
|
||
**API endpoints:**
|
||
```
|
||
GET /api/utterances # list with filters: date, room_id, speaker_id, match_status
|
||
GET /api/utterances/{id}/clip # stream audio clip WAV
|
||
POST /api/utterances/{id}/tag # label utterance: {speaker_id} or {name, create_new: true}
|
||
POST /api/utterances/{id}/dismiss # mark as noise/unknown, no review needed
|
||
GET /api/speakers # list all known speakers with utterance counts
|
||
POST /api/speakers # create new speaker
|
||
GET /api/rooms # list rooms with is_active, last_seen
|
||
POST /api/rooms/{id}/mute # send mute command to room's WebSocket connection
|
||
GET /api/search?q=... # full-text search across utterances.transcript (SQLite FTS5)
|
||
POST /api/ask # RAG query: builds context from utterances, calls Ollama
|
||
GET /api/summary/{date} # fetch or trigger daily summary for a date
|
||
```
|
||
|
||
**UI screens (5 tabs, single-page app):**
|
||
1. **Daily Log** — timeline grouped by time blocks, color-coded by speaker, inline Play/Tag/Dismiss on unknowns
|
||
2. **Search** — full-text with keyword highlighting, date-range filter
|
||
3. **Speakers** — known speaker cards + review-queue banner for unreviewed utterances
|
||
4. **Ask AI** — RAG chat backed by Ollama, responses cite room + timestamp sources
|
||
5. **Rooms** — live capture device status, per-room mute toggle, link to open capture page
|
||
|
||
### 5. Nightly Cron Jobs
|
||
Run at midnight via cron on hf-pdocker-01.
|
||
|
||
**Daily summary:** Fetch all utterances for the day → batch into prompt → send to Ollama (`qwen2.5:14b`) → store in `daily_summaries`. Prompt instructs the model to summarize key decisions, activities, and topics — not to transcribe verbatim.
|
||
|
||
**Unknown speaker clustering:** Load all `unknown` and `ambiguous` utterance embeddings from the past 7 days → cluster by cosine similarity (threshold 0.70) → group consistent voices as `Unknown #1`, `Unknown #2`, etc. in the review queue rather than scattered individuals.
|
||
|
||
---
|
||
|
||
## Data Model
|
||
|
||
SQLite at `/data/linkedstorm/linkedstorm.db`. WAL mode, foreign keys enabled. Thread-local connections (same pattern as frigate-notify).
|
||
|
||
```sql
|
||
CREATE TABLE rooms (
|
||
id INTEGER PRIMARY KEY,
|
||
name TEXT NOT NULL,
|
||
device_label TEXT,
|
||
is_active BOOLEAN DEFAULT 0,
|
||
last_seen DATETIME,
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
|
||
CREATE TABLE speakers (
|
||
id INTEGER PRIMARY KEY,
|
||
name TEXT NOT NULL,
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
|
||
CREATE TABLE voice_embeddings (
|
||
id INTEGER PRIMARY KEY,
|
||
speaker_id INTEGER NOT NULL REFERENCES speakers(id),
|
||
embedding BLOB NOT NULL, -- packed float32
|
||
audio_sample BLOB, -- short reference WAV
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
|
||
CREATE TABLE utterances (
|
||
id INTEGER PRIMARY KEY,
|
||
room_id INTEGER NOT NULL REFERENCES rooms(id),
|
||
speaker_id INTEGER REFERENCES speakers(id), -- nullable
|
||
transcript TEXT NOT NULL,
|
||
embedding BLOB, -- packed float32
|
||
match_status TEXT DEFAULT 'unreviewed', -- known/unknown/ambiguous/unreviewed
|
||
match_confidence REAL,
|
||
start_time DATETIME NOT NULL,
|
||
end_time DATETIME NOT NULL,
|
||
audio_clip_path TEXT,
|
||
livekit_room TEXT,
|
||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
|
||
CREATE VIRTUAL TABLE utterances_fts USING fts5(
|
||
transcript, content=utterances, content_rowid=id
|
||
);
|
||
|
||
CREATE TABLE daily_summaries (
|
||
id INTEGER PRIMARY KEY,
|
||
date DATE NOT NULL UNIQUE,
|
||
summary_text TEXT NOT NULL,
|
||
generated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||
);
|
||
```
|
||
|
||
---
|
||
|
||
## Speaker Identification
|
||
|
||
Mirrors frigate-notify's CLIP embedding pattern, adapted for voice:
|
||
|
||
| Similarity | Status | Action |
|
||
|---|---|---|
|
||
| ≥ 0.85 | `known` | Auto-label speaker. Store utterance embedding as additional reference. |
|
||
| 0.60–0.84 | `ambiguous` | Store utterance. Appear in review queue with Play + Tag buttons. |
|
||
| < 0.60 | `unknown` | Store utterance. Cluster with other unknowns nightly. |
|
||
| No embed | `unreviewed` | SpeechBrain unavailable. Transcript stored, embed missing. |
|
||
|
||
**Labeling flow (same as frigate-notify tag workflow):**
|
||
`POST /api/utterances/{id}/tag` with `{speaker_id}` or `{name, create_new: true}` →
|
||
sets `match_status='known'`, `match_confidence=1.0`, stores utterance's embedding in `voice_embeddings` for future matching.
|
||
|
||
---
|
||
|
||
## Deployment
|
||
|
||
**3 containers on hf-pdocker-01**
|
||
**Compose file:** `compose/hf-pdocker-01/linkedstorm/docker-compose.yml`
|
||
**Ports:**
|
||
- `8300` — `linkedstorm-pipeline` WebSocket ingestion (capture devices connect here)
|
||
- `8310` — `linkedstorm-api` FastAPI web UI and API
|
||
|
||
**Volumes:**
|
||
- `/data/linkedstorm/` → `/mnt/docker-data/configs/linkedstorm/` (NAS)
|
||
- `linkedstorm.db` — SQLite database
|
||
- `clips/` — audio clip WAV files
|
||
|
||
**Clip retention:** A VAD-triggered household produces roughly 50–200MB of WAV clips per day. The nightly cron should delete clips older than 30 days (configurable via `CLIP_RETENTION_DAYS` env var) while keeping the transcript and embedding rows permanently. The `audio_clip_path` column is set to `NULL` for purged clips; the Play button in the UI shows "clip expired" for those utterances.
|
||
|
||
**Environment:**
|
||
```
|
||
SPEACHES_URL=http://192.168.86.150:8000
|
||
CLIP_RETENTION_DAYS=30
|
||
OLLAMA_URL=http://192.168.86.150:11434
|
||
LIVEKIT_URL=ws://192.168.86.150:7880
|
||
LIVEKIT_API_KEY=...
|
||
LIVEKIT_API_SECRET=...
|
||
```
|
||
|
||
**Traefik:** `linkedstorm.hobbs.farm` → port 8310 (dashboard). Capture page accessed at `linkedstorm.hobbs.farm/capture?room=Kitchen`.
|
||
|
||
**Cron entries on hf-pdocker-01:**
|
||
```
|
||
0 0 * * * docker exec linkedstorm python -m cron.daily_summary
|
||
5 0 * * * docker exec linkedstorm python -m cron.cluster_unknowns
|
||
10 0 * * * docker exec linkedstorm python -m cron.purge_clips
|
||
```
|
||
|
||
---
|
||
|
||
## File Structure
|
||
|
||
```
|
||
compose/hf-pdocker-01/linkedstorm/
|
||
├── docker-compose.yml
|
||
├── .env.example
|
||
├── pipeline/
|
||
│ └── Dockerfile # linkedstorm-pipeline image
|
||
├── api/
|
||
│ └── Dockerfile # linkedstorm-api image
|
||
└── cron/
|
||
└── Dockerfile # linkedstorm-cron image
|
||
|
||
# Shared app code (mounted into pipeline + api containers, or baked into images)
|
||
app/
|
||
├── config.py # Settings from env vars (shared)
|
||
├── database.py # SQLite thread-local connections, schema init (shared)
|
||
├── models.py # Dataclasses for rooms, speakers, utterances (shared)
|
||
│
|
||
├── pipeline/ # linkedstorm-pipeline
|
||
│ ├── main.py # asyncio entrypoint: WebSocket server + per-room VAD loops
|
||
│ ├── ingestion.py # WebSocket handler, per-room audio buffer + Silero VAD
|
||
│ ├── processor.py # process_utterance: Whisper + resemblyzer + matching + persist
|
||
│ ├── recognition.py # resemblyzer embed + cosine similarity, speaker matching
|
||
│ └── livekit.py # Publish audio segments to LiveKit rooms
|
||
│
|
||
├── api/ # linkedstorm-api
|
||
│ ├── main.py # FastAPI entrypoint, mounts routes, serves static files
|
||
│ ├── routes/
|
||
│ │ ├── utterances.py # /api/utterances CRUD + tag + dismiss
|
||
│ │ ├── speakers.py # /api/speakers CRUD
|
||
│ │ ├── rooms.py # /api/rooms status + mute relay to pipeline
|
||
│ │ ├── search.py # /api/search (SQLite FTS5)
|
||
│ │ └── rag.py # /api/ask, /api/summary
|
||
│ └── static/
|
||
│ ├── index.html # Dashboard SPA (5-tab layout)
|
||
│ ├── capture.html # Minimal capture page (one per room device)
|
||
│ └── app.js / style.css
|
||
│
|
||
└── cron/ # linkedstorm-cron
|
||
├── main.py # Sleep-loop entrypoint: runs all jobs at midnight
|
||
├── daily_summary.py # Ollama daily summary generation
|
||
├── cluster_unknowns.py # Nightly unknown speaker clustering
|
||
└── purge_clips.py # Delete WAV files older than CLIP_RETENTION_DAYS
|
||
```
|
||
|
||
---
|
||
|
||
## Verification
|
||
|
||
1. **Capture → ingestion:** Open `capture.html?room=Kitchen` in a browser, speak — confirm WebSocket connects, room appears active in the Rooms tab of the dashboard
|
||
2. **VAD:** Speak briefly then stay silent — confirm a single utterance appears in the log (not multiple fragments), and background noise is not logged
|
||
3. **Transcription:** Read a known sentence — confirm transcript text in the utterance matches
|
||
4. **Speaker labeling:** Tag an unknown utterance as "Jeremy" → speak again → confirm next utterances auto-resolve to Jeremy with high confidence
|
||
5. **Mute:** Click mute on the capture page — confirm no utterances logged while muted, muted status shown in Rooms tab
|
||
6. **Search:** Search for a word spoken earlier — confirm it appears highlighted in results
|
||
7. **Ask AI:** Ask "What did we talk about today?" — confirm Ollama returns a coherent summary with room/time citations
|
||
8. **LiveKit replay:** From the dashboard, join a room's LiveKit channel — confirm processed audio plays back
|
||
9. **Nightly cron:** Trigger `daily_summary` manually — confirm `daily_summaries` row appears and is shown in the log view
|
||
10. **Prometheus probe:** Add `http://192.168.86.192:8310/health` to blackbox targets and confirm it's green
|