hobbs-iac/docs/superpowers/specs/2026-05-27-linkedstorm-design.md

LinkedStorm — Home Audio Assistant Design

Date: 2026-05-27
Status: Approved

---

Context

The goal is a whole-home audio assistant that passively records household conversations, identifies who is speaking, produces searchable transcripts, and maintains a daily activity log. The motivation is to have a reliable, private, locally-hosted record of decisions and conversations that can be queried naturally ("what did we decide about the vacation?") without relying on cloud services.

The system reuses as much existing homelab infrastructure as possible: Whisper STT on the GPU machine, LiveKit for WebRTC, Ollama for LLM queries, and SpeechBrain for audio ML — all already deployed. Speaker labeling mirrors the proven frigate-notify person-identification pattern (CLIP embeddings → voice embeddings, cosine similarity, three-tier matching, review-and-tag UI).

---

Architecture

Hybrid approach: Lightweight WebSocket audio capture from browser tabs → server-side processing pipeline → LiveKit for live monitoring and replay.

Container Breakdown

Every deployable component is its own container, following the same pattern as VoiceCoach (7 containers) and the rest of the homelab. LinkedStorm is 3 containers:

Container	Purpose	Port	Host
linkedstorm-pipeline	WebSocket ingestion, Silero VAD, Whisper calls, resemblyzer speaker embedding, cosine similarity matching, SQLite writes, LiveKit publish	8300	hf-pdocker-01
linkedstorm-api	FastAPI web dashboard + REST API, static UI, FTS5 search, RAG queries, speaker labeling	8310	hf-pdocker-01
linkedstorm-cron	Nightly Ollama summary, unknown speaker clustering, WAV clip purge	— (no port)	hf-pdocker-01

All three share a single Docker volume for the SQLite database and clips directory.

External Services (Reused Without Modification)

Service	What	Where
Speaches/Whisper	POST /v1/audio/transcriptions	GPU machine :8000
LiveKit	Per-room replay rooms (linkedstorm-{room})	GPU machine :7880
Ollama	RAG queries + daily summaries (qwen2.5:14b)	GPU machine :11434

Speaker voice embedding uses resemblyzer (ECAPA-TDNN, ~50MB) running in-process inside linkedstorm-pipeline — no separate microservice, no coupling to VoiceCoach.

Data Flow

Browser tab (capture.html)
  └─ MediaRecorder → WebSocket → linkedstorm-pipeline:8300
       └─ Silero VAD detects speech segment
            └─ [Whisper STT via Speaches ‖ resemblyzer embed] concurrent
                 └─ cosine similarity → speaker match
                      └─ SQLite write + WAV clip save
                           └─ LiveKit publish → dashboard replay
linkedstorm-api:8310 ← reads SQLite → serves dashboard + API
linkedstorm-cron ← runs nightly at midnight via entrypoint sleep loop

---

Components

1. Browser Capture Page (linkedstorm-capture)

Single HTML page opened on any device to make it a room microphone.

• MediaRecorder at 16kHz mono, 250ms chunks streamed over WebSocket
• Room name set via URL param (?room=Kitchen) or prompt on first load
• Persistent room name stored in localStorage
• Mute toggle: stops MediaRecorder, sends {"type":"mute"} control message, server discards buffered audio for that room
• Visual indicator: green pulsing dot (streaming) / red dot (muted) / gray (disconnected)
• Auto-reconnect with exponential backoff on WebSocket drop

2. Audio Ingestion Service (linkedstorm-ingestion)

Python asyncio service on hf-pdocker-01. Single process, per-room coroutines.

• WebSocket endpoint: ws://linkedstorm:8300/ws/{room_id}
• Per-room asyncio.Queue of audio chunks
• Silero VAD (reused from VoiceCoach): processes each 250ms chunk; accumulates on speech-detected, emits segment on ≥600ms silence; discards segments <0.5s
• On segment emit: calls process_utterance(room_id, audio_bytes, start_time, end_time)
• Room registry: upsert rooms table on first connection, update last_seen on each chunk, set is_active=False on disconnect

3. Processing Pipeline

Runs as an async task per emitted speech segment.

Step 1 — Concurrent inference (asyncio gather):

• POST http://192.168.86.150:8000/v1/audio/transcriptions with audio WAV → transcript text
• resemblyzer.preprocess_wav() + encoder.embed_utterance() in-process → voice embedding (256-dim float32 array)

Step 2 — Speaker matching:

• Load all voice_embeddings rows from SQLite
• Compute cosine similarity between utterance embedding and each reference
• Group by speaker_id, take MAX per speaker
• Thresholds: ≥0.85 → known, 0.60–0.84 → ambiguous, <0.60 → unknown

Step 3 — Persist:

• Save audio clip to /data/linkedstorm/clips/{date}/{room_id}/{utterance_id}.wav
• Insert utterances row: transcript, embedding, room_id, speaker_id, match_status, match_confidence, start_time, end_time, audio_clip_path

Step 4 — LiveKit publish:

• Publish audio segment to LiveKit room linkedstorm-{room_id} for live monitoring/replay

4. Web UI Service (linkedstorm-ui)

FastAPI on hf-pdocker-01 port 8310. Static HTML/JS frontend, JSON API backend. Same pattern as frigate-notify.

API endpoints:

GET  /api/utterances          # list with filters: date, room_id, speaker_id, match_status
GET  /api/utterances/{id}/clip  # stream audio clip WAV
POST /api/utterances/{id}/tag   # label utterance: {speaker_id} or {name, create_new: true}
POST /api/utterances/{id}/dismiss  # mark as noise/unknown, no review needed
GET  /api/speakers            # list all known speakers with utterance counts
POST /api/speakers            # create new speaker
GET  /api/rooms               # list rooms with is_active, last_seen
POST /api/rooms/{id}/mute     # send mute command to room's WebSocket connection
GET  /api/search?q=...        # full-text search across utterances.transcript (SQLite FTS5)
POST /api/ask                 # RAG query: builds context from utterances, calls Ollama
GET  /api/summary/{date}      # fetch or trigger daily summary for a date

UI screens (5 tabs, single-page app):

1. Daily Log — timeline grouped by time blocks, color-coded by speaker, inline Play/Tag/Dismiss on unknowns
2. Search — full-text with keyword highlighting, date-range filter
3. Speakers — known speaker cards + review-queue banner for unreviewed utterances
4. Ask AI — RAG chat backed by Ollama, responses cite room + timestamp sources
5. Rooms — live capture device status, per-room mute toggle, link to open capture page

5. Nightly Cron Jobs

Run at midnight via cron on hf-pdocker-01.

Daily summary: Fetch all utterances for the day → batch into prompt → send to Ollama (qwen2.5:14b) → store in daily_summaries. Prompt instructs the model to summarize key decisions, activities, and topics — not to transcribe verbatim.

Unknown speaker clustering: Load all unknown and ambiguous utterance embeddings from the past 7 days → cluster by cosine similarity (threshold 0.70) → group consistent voices as Unknown #1, Unknown #2, etc. in the review queue rather than scattered individuals.

---

Data Model

SQLite at /data/linkedstorm/linkedstorm.db. WAL mode, foreign keys enabled. Thread-local connections (same pattern as frigate-notify).

CREATE TABLE rooms (
  id          INTEGER PRIMARY KEY,
  name        TEXT NOT NULL,
  device_label TEXT,
  is_active   BOOLEAN DEFAULT 0,
  last_seen   DATETIME,
  created_at  DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE speakers (
  id          INTEGER PRIMARY KEY,
  name        TEXT NOT NULL,
  created_at  DATETIME DEFAULT CURRENT_TIMESTAMP,
  updated_at  DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE voice_embeddings (
  id          INTEGER PRIMARY KEY,
  speaker_id  INTEGER NOT NULL REFERENCES speakers(id),
  embedding   BLOB NOT NULL,          -- packed float32
  audio_sample BLOB,                  -- short reference WAV
  created_at  DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE utterances (
  id                INTEGER PRIMARY KEY,
  room_id           INTEGER NOT NULL REFERENCES rooms(id),
  speaker_id        INTEGER REFERENCES speakers(id),  -- nullable
  transcript        TEXT NOT NULL,
  embedding         BLOB,                             -- packed float32
  match_status      TEXT DEFAULT 'unreviewed',        -- known/unknown/ambiguous/unreviewed
  match_confidence  REAL,
  start_time        DATETIME NOT NULL,
  end_time          DATETIME NOT NULL,
  audio_clip_path   TEXT,
  livekit_room      TEXT,
  created_at        DATETIME DEFAULT CURRENT_TIMESTAMP
);

CREATE VIRTUAL TABLE utterances_fts USING fts5(
  transcript, content=utterances, content_rowid=id
);

CREATE TABLE daily_summaries (
  id            INTEGER PRIMARY KEY,
  date          DATE NOT NULL UNIQUE,
  summary_text  TEXT NOT NULL,
  generated_at  DATETIME DEFAULT CURRENT_TIMESTAMP
);

---

Speaker Identification

Mirrors frigate-notify's CLIP embedding pattern, adapted for voice:

Similarity	Status	Action
≥ 0.85	known	Auto-label speaker. Store utterance embedding as additional reference.
0.60–0.84	ambiguous	Store utterance. Appear in review queue with Play + Tag buttons.
< 0.60	unknown	Store utterance. Cluster with other unknowns nightly.
No embed	unreviewed	SpeechBrain unavailable. Transcript stored, embed missing.

Labeling flow (same as frigate-notify tag workflow): POST /api/utterances/{id}/tag with {speaker_id} or {name, create_new: true} → sets match_status='known', match_confidence=1.0, stores utterance's embedding in voice_embeddings for future matching.

---

Deployment

3 containers on hf-pdocker-01
Compose file: compose/hf-pdocker-01/linkedstorm/docker-compose.yml
Ports:

• 8300 — linkedstorm-pipeline WebSocket ingestion (capture devices connect here)
• 8310 — linkedstorm-api FastAPI web UI and API

Volumes:

• /data/linkedstorm/ → /mnt/docker-data/configs/linkedstorm/ (NAS)
  • linkedstorm.db — SQLite database
  • clips/ — audio clip WAV files

Clip retention: A VAD-triggered household produces roughly 50–200MB of WAV clips per day. The nightly cron should delete clips older than 30 days (configurable via CLIP_RETENTION_DAYS env var) while keeping the transcript and embedding rows permanently. The audio_clip_path column is set to NULL for purged clips; the Play button in the UI shows "clip expired" for those utterances.

Environment:

SPEACHES_URL=http://192.168.86.150:8000
CLIP_RETENTION_DAYS=30
OLLAMA_URL=http://192.168.86.150:11434
LIVEKIT_URL=ws://192.168.86.150:7880
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...

Traefik: linkedstorm.hobbs.farm → port 8310 (dashboard). Capture page accessed at linkedstorm.hobbs.farm/capture?room=Kitchen.

Cron entries on hf-pdocker-01:

0 0 * * * docker exec linkedstorm python -m cron.daily_summary
5 0 * * * docker exec linkedstorm python -m cron.cluster_unknowns
10 0 * * * docker exec linkedstorm python -m cron.purge_clips

---

File Structure

compose/hf-pdocker-01/linkedstorm/
├── docker-compose.yml
├── .env.example
├── pipeline/
│   └── Dockerfile        # linkedstorm-pipeline image
├── api/
│   └── Dockerfile        # linkedstorm-api image
└── cron/
    └── Dockerfile        # linkedstorm-cron image

# Shared app code (mounted into pipeline + api containers, or baked into images)
app/
├── config.py             # Settings from env vars (shared)
├── database.py           # SQLite thread-local connections, schema init (shared)
├── models.py             # Dataclasses for rooms, speakers, utterances (shared)
│
├── pipeline/             # linkedstorm-pipeline
│   ├── main.py           # asyncio entrypoint: WebSocket server + per-room VAD loops
│   ├── ingestion.py      # WebSocket handler, per-room audio buffer + Silero VAD
│   ├── processor.py      # process_utterance: Whisper + resemblyzer + matching + persist
│   ├── recognition.py    # resemblyzer embed + cosine similarity, speaker matching
│   └── livekit.py        # Publish audio segments to LiveKit rooms
│
├── api/                  # linkedstorm-api
│   ├── main.py           # FastAPI entrypoint, mounts routes, serves static files
│   ├── routes/
│   │   ├── utterances.py # /api/utterances CRUD + tag + dismiss
│   │   ├── speakers.py   # /api/speakers CRUD
│   │   ├── rooms.py      # /api/rooms status + mute relay to pipeline
│   │   ├── search.py     # /api/search (SQLite FTS5)
│   │   └── rag.py        # /api/ask, /api/summary
│   └── static/
│       ├── index.html    # Dashboard SPA (5-tab layout)
│       ├── capture.html  # Minimal capture page (one per room device)
│       └── app.js / style.css
│
└── cron/                 # linkedstorm-cron
    ├── main.py           # Sleep-loop entrypoint: runs all jobs at midnight
    ├── daily_summary.py  # Ollama daily summary generation
    ├── cluster_unknowns.py # Nightly unknown speaker clustering
    └── purge_clips.py    # Delete WAV files older than CLIP_RETENTION_DAYS

---

Verification

1. Capture → ingestion: Open capture.html?room=Kitchen in a browser, speak — confirm WebSocket connects, room appears active in the Rooms tab of the dashboard
2. VAD: Speak briefly then stay silent — confirm a single utterance appears in the log (not multiple fragments), and background noise is not logged
3. Transcription: Read a known sentence — confirm transcript text in the utterance matches
4. Speaker labeling: Tag an unknown utterance as "Jeremy" → speak again → confirm next utterances auto-resolve to Jeremy with high confidence
5. Mute: Click mute on the capture page — confirm no utterances logged while muted, muted status shown in Rooms tab
6. Search: Search for a word spoken earlier — confirm it appears highlighted in results
7. Ask AI: Ask "What did we talk about today?" — confirm Ollama returns a coherent summary with room/time citations
8. LiveKit replay: From the dashboard, join a room's LiveKit channel — confirm processed audio plays back
9. Nightly cron: Trigger daily_summary manually — confirm daily_summaries row appears and is shown in the log view
10. Prometheus probe: Add http://192.168.86.192:8310/health to blackbox targets and confirm it's green