docs: add LinkedStorm home audio assistant design spec
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,303 @@
|
||||
# LinkedStorm — Home Audio Assistant Design
|
||||
|
||||
**Date:** 2026-05-27
|
||||
**Status:** Approved
|
||||
|
||||
---
|
||||
|
||||
## Context
|
||||
|
||||
The goal is a whole-home audio assistant that passively records household conversations, identifies who is speaking, produces searchable transcripts, and maintains a daily activity log. The motivation is to have a reliable, private, locally-hosted record of decisions and conversations that can be queried naturally ("what did we decide about the vacation?") without relying on cloud services.
|
||||
|
||||
The system reuses as much existing homelab infrastructure as possible: Whisper STT on the GPU machine, LiveKit for WebRTC, Ollama for LLM queries, and SpeechBrain for audio ML — all already deployed. Speaker labeling mirrors the proven frigate-notify person-identification pattern (CLIP embeddings → voice embeddings, cosine similarity, three-tier matching, review-and-tag UI).
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
**Hybrid approach:** Lightweight WebSocket audio capture from browser tabs → server-side processing pipeline → LiveKit for live monitoring and replay.
|
||||
|
||||
### Container Breakdown
|
||||
|
||||
Every deployable component is its own container, following the same pattern as VoiceCoach (7 containers) and the rest of the homelab. LinkedStorm is 3 containers:
|
||||
|
||||
| Container | Purpose | Port | Host |
|
||||
|---|---|---|---|
|
||||
| `linkedstorm-pipeline` | WebSocket ingestion, Silero VAD, Whisper calls, resemblyzer speaker embedding, cosine similarity matching, SQLite writes, LiveKit publish | 8300 | hf-pdocker-01 |
|
||||
| `linkedstorm-api` | FastAPI web dashboard + REST API, static UI, FTS5 search, RAG queries, speaker labeling | 8310 | hf-pdocker-01 |
|
||||
| `linkedstorm-cron` | Nightly Ollama summary, unknown speaker clustering, WAV clip purge | — (no port) | hf-pdocker-01 |
|
||||
|
||||
All three share a single Docker volume for the SQLite database and clips directory.
|
||||
|
||||
### External Services (Reused Without Modification)
|
||||
|
||||
| Service | What | Where |
|
||||
|---|---|---|
|
||||
| Speaches/Whisper | `POST /v1/audio/transcriptions` | GPU machine :8000 |
|
||||
| LiveKit | Per-room replay rooms (`linkedstorm-{room}`) | GPU machine :7880 |
|
||||
| Ollama | RAG queries + daily summaries (`qwen2.5:14b`) | GPU machine :11434 |
|
||||
|
||||
Speaker voice embedding uses `resemblyzer` (ECAPA-TDNN, ~50MB) running in-process inside `linkedstorm-pipeline` — no separate microservice, no coupling to VoiceCoach.
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
Browser tab (capture.html)
|
||||
└─ MediaRecorder → WebSocket → linkedstorm-pipeline:8300
|
||||
└─ Silero VAD detects speech segment
|
||||
└─ [Whisper STT via Speaches ‖ resemblyzer embed] concurrent
|
||||
└─ cosine similarity → speaker match
|
||||
└─ SQLite write + WAV clip save
|
||||
└─ LiveKit publish → dashboard replay
|
||||
linkedstorm-api:8310 ← reads SQLite → serves dashboard + API
|
||||
linkedstorm-cron ← runs nightly at midnight via entrypoint sleep loop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Browser Capture Page (`linkedstorm-capture`)
|
||||
Single HTML page opened on any device to make it a room microphone.
|
||||
|
||||
- `MediaRecorder` at 16kHz mono, 250ms chunks streamed over WebSocket
|
||||
- Room name set via URL param (`?room=Kitchen`) or prompt on first load
|
||||
- Persistent room name stored in `localStorage`
|
||||
- Mute toggle: stops `MediaRecorder`, sends `{"type":"mute"}` control message, server discards buffered audio for that room
|
||||
- Visual indicator: green pulsing dot (streaming) / red dot (muted) / gray (disconnected)
|
||||
- Auto-reconnect with exponential backoff on WebSocket drop
|
||||
|
||||
### 2. Audio Ingestion Service (`linkedstorm-ingestion`)
|
||||
Python asyncio service on hf-pdocker-01. Single process, per-room coroutines.
|
||||
|
||||
- WebSocket endpoint: `ws://linkedstorm:8300/ws/{room_id}`
|
||||
- Per-room `asyncio.Queue` of audio chunks
|
||||
- **Silero VAD** (reused from VoiceCoach): processes each 250ms chunk; accumulates on speech-detected, emits segment on ≥600ms silence; discards segments <0.5s
|
||||
- On segment emit: calls `process_utterance(room_id, audio_bytes, start_time, end_time)`
|
||||
- Room registry: upsert `rooms` table on first connection, update `last_seen` on each chunk, set `is_active=False` on disconnect
|
||||
|
||||
### 3. Processing Pipeline
|
||||
Runs as an async task per emitted speech segment.
|
||||
|
||||
**Step 1 — Concurrent inference** (asyncio gather):
|
||||
- `POST http://192.168.86.150:8000/v1/audio/transcriptions` with audio WAV → transcript text
|
||||
- `resemblyzer.preprocess_wav()` + `encoder.embed_utterance()` in-process → voice embedding (256-dim float32 array)
|
||||
|
||||
**Step 2 — Speaker matching:**
|
||||
- Load all `voice_embeddings` rows from SQLite
|
||||
- Compute cosine similarity between utterance embedding and each reference
|
||||
- Group by `speaker_id`, take MAX per speaker
|
||||
- Thresholds: ≥0.85 → `known`, 0.60–0.84 → `ambiguous`, <0.60 → `unknown`
|
||||
|
||||
**Step 3 — Persist:**
|
||||
- Save audio clip to `/data/linkedstorm/clips/{date}/{room_id}/{utterance_id}.wav`
|
||||
- Insert `utterances` row: transcript, embedding, room_id, speaker_id, match_status, match_confidence, start_time, end_time, audio_clip_path
|
||||
|
||||
**Step 4 — LiveKit publish:**
|
||||
- Publish audio segment to LiveKit room `linkedstorm-{room_id}` for live monitoring/replay
|
||||
|
||||
### 4. Web UI Service (`linkedstorm-ui`)
|
||||
FastAPI on hf-pdocker-01 port 8310. Static HTML/JS frontend, JSON API backend. Same pattern as frigate-notify.
|
||||
|
||||
**API endpoints:**
|
||||
```
|
||||
GET /api/utterances # list with filters: date, room_id, speaker_id, match_status
|
||||
GET /api/utterances/{id}/clip # stream audio clip WAV
|
||||
POST /api/utterances/{id}/tag # label utterance: {speaker_id} or {name, create_new: true}
|
||||
POST /api/utterances/{id}/dismiss # mark as noise/unknown, no review needed
|
||||
GET /api/speakers # list all known speakers with utterance counts
|
||||
POST /api/speakers # create new speaker
|
||||
GET /api/rooms # list rooms with is_active, last_seen
|
||||
POST /api/rooms/{id}/mute # send mute command to room's WebSocket connection
|
||||
GET /api/search?q=... # full-text search across utterances.transcript (SQLite FTS5)
|
||||
POST /api/ask # RAG query: builds context from utterances, calls Ollama
|
||||
GET /api/summary/{date} # fetch or trigger daily summary for a date
|
||||
```
|
||||
|
||||
**UI screens (5 tabs, single-page app):**
|
||||
1. **Daily Log** — timeline grouped by time blocks, color-coded by speaker, inline Play/Tag/Dismiss on unknowns
|
||||
2. **Search** — full-text with keyword highlighting, date-range filter
|
||||
3. **Speakers** — known speaker cards + review-queue banner for unreviewed utterances
|
||||
4. **Ask AI** — RAG chat backed by Ollama, responses cite room + timestamp sources
|
||||
5. **Rooms** — live capture device status, per-room mute toggle, link to open capture page
|
||||
|
||||
### 5. Nightly Cron Jobs
|
||||
Run at midnight via cron on hf-pdocker-01.
|
||||
|
||||
**Daily summary:** Fetch all utterances for the day → batch into prompt → send to Ollama (`qwen2.5:14b`) → store in `daily_summaries`. Prompt instructs the model to summarize key decisions, activities, and topics — not to transcribe verbatim.
|
||||
|
||||
**Unknown speaker clustering:** Load all `unknown` and `ambiguous` utterance embeddings from the past 7 days → cluster by cosine similarity (threshold 0.70) → group consistent voices as `Unknown #1`, `Unknown #2`, etc. in the review queue rather than scattered individuals.
|
||||
|
||||
---
|
||||
|
||||
## Data Model
|
||||
|
||||
SQLite at `/data/linkedstorm/linkedstorm.db`. WAL mode, foreign keys enabled. Thread-local connections (same pattern as frigate-notify).
|
||||
|
||||
```sql
|
||||
CREATE TABLE rooms (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
device_label TEXT,
|
||||
is_active BOOLEAN DEFAULT 0,
|
||||
last_seen DATETIME,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE speakers (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE voice_embeddings (
|
||||
id INTEGER PRIMARY KEY,
|
||||
speaker_id INTEGER NOT NULL REFERENCES speakers(id),
|
||||
embedding BLOB NOT NULL, -- packed float32
|
||||
audio_sample BLOB, -- short reference WAV
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE TABLE utterances (
|
||||
id INTEGER PRIMARY KEY,
|
||||
room_id INTEGER NOT NULL REFERENCES rooms(id),
|
||||
speaker_id INTEGER REFERENCES speakers(id), -- nullable
|
||||
transcript TEXT NOT NULL,
|
||||
embedding BLOB, -- packed float32
|
||||
match_status TEXT DEFAULT 'unreviewed', -- known/unknown/ambiguous/unreviewed
|
||||
match_confidence REAL,
|
||||
start_time DATETIME NOT NULL,
|
||||
end_time DATETIME NOT NULL,
|
||||
audio_clip_path TEXT,
|
||||
livekit_room TEXT,
|
||||
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE VIRTUAL TABLE utterances_fts USING fts5(
|
||||
transcript, content=utterances, content_rowid=id
|
||||
);
|
||||
|
||||
CREATE TABLE daily_summaries (
|
||||
id INTEGER PRIMARY KEY,
|
||||
date DATE NOT NULL UNIQUE,
|
||||
summary_text TEXT NOT NULL,
|
||||
generated_at DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Speaker Identification
|
||||
|
||||
Mirrors frigate-notify's CLIP embedding pattern, adapted for voice:
|
||||
|
||||
| Similarity | Status | Action |
|
||||
|---|---|---|
|
||||
| ≥ 0.85 | `known` | Auto-label speaker. Store utterance embedding as additional reference. |
|
||||
| 0.60–0.84 | `ambiguous` | Store utterance. Appear in review queue with Play + Tag buttons. |
|
||||
| < 0.60 | `unknown` | Store utterance. Cluster with other unknowns nightly. |
|
||||
| No embed | `unreviewed` | SpeechBrain unavailable. Transcript stored, embed missing. |
|
||||
|
||||
**Labeling flow (same as frigate-notify tag workflow):**
|
||||
`POST /api/utterances/{id}/tag` with `{speaker_id}` or `{name, create_new: true}` →
|
||||
sets `match_status='known'`, `match_confidence=1.0`, stores utterance's embedding in `voice_embeddings` for future matching.
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
**3 containers on hf-pdocker-01**
|
||||
**Compose file:** `compose/hf-pdocker-01/linkedstorm/docker-compose.yml`
|
||||
**Ports:**
|
||||
- `8300` — `linkedstorm-pipeline` WebSocket ingestion (capture devices connect here)
|
||||
- `8310` — `linkedstorm-api` FastAPI web UI and API
|
||||
|
||||
**Volumes:**
|
||||
- `/data/linkedstorm/` → `/mnt/docker-data/configs/linkedstorm/` (NAS)
|
||||
- `linkedstorm.db` — SQLite database
|
||||
- `clips/` — audio clip WAV files
|
||||
|
||||
**Clip retention:** A VAD-triggered household produces roughly 50–200MB of WAV clips per day. The nightly cron should delete clips older than 30 days (configurable via `CLIP_RETENTION_DAYS` env var) while keeping the transcript and embedding rows permanently. The `audio_clip_path` column is set to `NULL` for purged clips; the Play button in the UI shows "clip expired" for those utterances.
|
||||
|
||||
**Environment:**
|
||||
```
|
||||
SPEACHES_URL=http://192.168.86.150:8000
|
||||
CLIP_RETENTION_DAYS=30
|
||||
OLLAMA_URL=http://192.168.86.150:11434
|
||||
LIVEKIT_URL=ws://192.168.86.150:7880
|
||||
LIVEKIT_API_KEY=...
|
||||
LIVEKIT_API_SECRET=...
|
||||
```
|
||||
|
||||
**Traefik:** `linkedstorm.hobbs.farm` → port 8310 (dashboard). Capture page accessed at `linkedstorm.hobbs.farm/capture?room=Kitchen`.
|
||||
|
||||
**Cron entries on hf-pdocker-01:**
|
||||
```
|
||||
0 0 * * * docker exec linkedstorm python -m cron.daily_summary
|
||||
5 0 * * * docker exec linkedstorm python -m cron.cluster_unknowns
|
||||
10 0 * * * docker exec linkedstorm python -m cron.purge_clips
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
compose/hf-pdocker-01/linkedstorm/
|
||||
├── docker-compose.yml
|
||||
├── .env.example
|
||||
├── pipeline/
|
||||
│ └── Dockerfile # linkedstorm-pipeline image
|
||||
├── api/
|
||||
│ └── Dockerfile # linkedstorm-api image
|
||||
└── cron/
|
||||
└── Dockerfile # linkedstorm-cron image
|
||||
|
||||
# Shared app code (mounted into pipeline + api containers, or baked into images)
|
||||
app/
|
||||
├── config.py # Settings from env vars (shared)
|
||||
├── database.py # SQLite thread-local connections, schema init (shared)
|
||||
├── models.py # Dataclasses for rooms, speakers, utterances (shared)
|
||||
│
|
||||
├── pipeline/ # linkedstorm-pipeline
|
||||
│ ├── main.py # asyncio entrypoint: WebSocket server + per-room VAD loops
|
||||
│ ├── ingestion.py # WebSocket handler, per-room audio buffer + Silero VAD
|
||||
│ ├── processor.py # process_utterance: Whisper + resemblyzer + matching + persist
|
||||
│ ├── recognition.py # resemblyzer embed + cosine similarity, speaker matching
|
||||
│ └── livekit.py # Publish audio segments to LiveKit rooms
|
||||
│
|
||||
├── api/ # linkedstorm-api
|
||||
│ ├── main.py # FastAPI entrypoint, mounts routes, serves static files
|
||||
│ ├── routes/
|
||||
│ │ ├── utterances.py # /api/utterances CRUD + tag + dismiss
|
||||
│ │ ├── speakers.py # /api/speakers CRUD
|
||||
│ │ ├── rooms.py # /api/rooms status + mute relay to pipeline
|
||||
│ │ ├── search.py # /api/search (SQLite FTS5)
|
||||
│ │ └── rag.py # /api/ask, /api/summary
|
||||
│ └── static/
|
||||
│ ├── index.html # Dashboard SPA (5-tab layout)
|
||||
│ ├── capture.html # Minimal capture page (one per room device)
|
||||
│ └── app.js / style.css
|
||||
│
|
||||
└── cron/ # linkedstorm-cron
|
||||
├── main.py # Sleep-loop entrypoint: runs all jobs at midnight
|
||||
├── daily_summary.py # Ollama daily summary generation
|
||||
├── cluster_unknowns.py # Nightly unknown speaker clustering
|
||||
└── purge_clips.py # Delete WAV files older than CLIP_RETENTION_DAYS
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
1. **Capture → ingestion:** Open `capture.html?room=Kitchen` in a browser, speak — confirm WebSocket connects, room appears active in the Rooms tab of the dashboard
|
||||
2. **VAD:** Speak briefly then stay silent — confirm a single utterance appears in the log (not multiple fragments), and background noise is not logged
|
||||
3. **Transcription:** Read a known sentence — confirm transcript text in the utterance matches
|
||||
4. **Speaker labeling:** Tag an unknown utterance as "Jeremy" → speak again → confirm next utterances auto-resolve to Jeremy with high confidence
|
||||
5. **Mute:** Click mute on the capture page — confirm no utterances logged while muted, muted status shown in Rooms tab
|
||||
6. **Search:** Search for a word spoken earlier — confirm it appears highlighted in results
|
||||
7. **Ask AI:** Ask "What did we talk about today?" — confirm Ollama returns a coherent summary with room/time citations
|
||||
8. **LiveKit replay:** From the dashboard, join a room's LiveKit channel — confirm processed audio plays back
|
||||
9. **Nightly cron:** Trigger `daily_summary` manually — confirm `daily_summaries` row appears and is shown in the log view
|
||||
10. **Prometheus probe:** Add `http://192.168.86.192:8310/health` to blackbox targets and confirm it's green
|
||||
Reference in New Issue
Block a user