Compare commits

14 Commits

Author SHA1 Message Date
jeremy e9a98f4855 chore: pin jellyfin to 2026060106
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-03 21:37:46 -05:00
jeremy 97e3cf12a7 frigate: sync config to live state, fix /tmp/cache size, enable VAAPI on house_east/pool_room, throttle shop_west detection
- docker-compose.yml: increase /tmp/cache tmpfs from 4G to 8G, add Coral USB
  device passthrough (/dev/bus/usb), fix memswap_limit to 16G (was equal to mem_limit)
- config.yml: replace OpenVINO detector with Coral edgetpu (usb), update model
  dimensions to 320x320 and pixel format to rgb, enable VAAPI on house_east and
  pool_room, throttle shop_west to 2fps with stricter person/car filters (min_area
  + threshold), add offline note for gate_camera, enable semantic_search (large),
  face_recognition (small), LPR, and bird classification

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-01 15:07:36 -05:00
jeremy 06e9db70d2 feat: replace legacy Unifi with UniFi OS Server v5.1.15
Deploys ghcr.io/lemker/unifi-os-server replacing linuxserver/unifi-network-application.
Data dirs moved to /var/lib/unifi-os/ locally (NFS root_squash blocks chown needed by
postgres/mongodb/nginx init). Only /persistent remains on NAS for config backups.
Access: https://192.168.86.100:11443

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 20:36:02 -05:00
jeremy b3bef75981 feat: VoiceCoach v2 — livekit-agents 1.5 personal voice assistant
Replaces pipecat 0.0.48 (6 manual patches, DuplicateIdentity reconnect storms)
with livekit-agents 1.5 SDK. New features:
- SvelteKit UI with live transcript, VAD ring, mode bar, vocal meter, history view
- SQLite session store with WAL mode (cross-session memory)
- Async SpeechBrain vocal/emotion analysis injected per turn
- Homelab tools: SSH/docker, Prometheus alerts, Home Assistant, SearXNG search
- OpenClaw LLM gateway preserved (qwen2.5:14b via Ollama)
- 4 modes: coaching, vent, practice, briefing

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-31 18:05:30 -05:00
jeremy 55f05a88d0 infra: migrate GPU workstation to Ubuntu, deploy vLLM-Hermes + voice stack
- Migrated primary workstation (192.168.86.27) from Windows/WSL2 to Ubuntu 24.04 (2026-05-28)
- Upgraded Ollama 0.15.6 → 0.24.0 (all 21 models preserved)
- Fixed Watchtower crash loop (DOCKER_API_VERSION=1.44)
- Fixed Open WebUI OLLAMA_BASE_URL to use Docker service name
- Deployed Speaches :8000 (CUDA, STT/TTS for VoiceCoach)
- Deployed LiveKit :7880/:7881 (WebRTC for VoiceCoach)
- Deployed vLLM-Hermes :8001 (Hermes 4.3 36B AWQ, tool-call-parser hermes)
  - Endpoint: http://192.168.86.27:8001/v1, model: hermes
  - Tool-call test passed: structured tool_calls array confirmed
- Added vllm-local provider to voicecoach-openclaw and margaret OpenClaw configs
- Added OPENAI_API_BASE_URL to Open WebUI for Hermes access in chat UI
- Updated CLAUDE.md: removed WSL2 references, added GPU workstation section

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-28 18:48:44 -05:00
jeremy 98cae1e08f feat: linkedstorm docker-compose and Traefik route
Add compose/hf-pdocker-01/linkedstorm/ with 3-container compose (pipeline :8300, api :8310, cron) and stacks/traefik/dynamic/linkedstorm.yml routing linkedstorm.hobbs.farm -> :8310.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-28 10:54:53 -05:00
jeremy cd3f3161da docs: add LinkedStorm home audio assistant design spec
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-27 21:59:32 -05:00
jeremy 904d1ef397 fix: onyx nginx - use rewrite to strip /api/ prefix with variable proxy_pass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 13:58:30 -05:00
jeremy a7dfbdeb79 fix: onyx nginx - use Docker DNS resolver to prevent stale upstream IPs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 13:30:42 -05:00
jeremy 6233cf9e1d fix: disable Onyx telemetry (stops data posting to telemetry.onyx.app) 2026-05-22 09:00:46 -05:00
jeremy 0d5b781aef fix: onyx nginx - strip /api prefix when proxying to api_server
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 08:30:37 -05:00
jeremy 6f485ff6e5 docs: update CLAUDE.md — Coral USB TPU, Onyx redeploy, infra fixes 2026-05-20
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-20 08:06:05 -05:00
jeremy 752b8230ed infra: clean redeploy of Onyx on bart with latest images
Full wipe and fresh start: removed gitea_patch/unstructured_patch
hacks from startup commands, upgraded all onyxdotapp images from
nightly-latest-20260331 pin to :latest, removed UNSTRUCTURED_SERVER_URL
(service decommissioned 2026-04-14), added Watchtower disable labels
to ALL services (was missing on web_server, model_servers, db, cache,
opensearch, vespa, nginx), added nginx.conf for reverse proxy.

Alembic ran 349 migrations cleanly on fresh DB. All 10 containers up,
0 restarts, HTTP 200 at :3005.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-20 07:12:03 -05:00
jeremy 03b7a46a96 fix: pin guacamole/guacd to 1.6.0 and disable Watchtower to prevent schema breakage
Watchtower pulled a newer guacamole:latest image whose JDBC extension expected
tables not present in the existing DB (guacamole_user_attribute etc.). Fixed by
reinitializing the DB with the official 1.6.0 schema (001-create-schema.sql +
002-create-admin-user.sql) and pinning both guacamole and guacd to 1.6.0 with
com.centurylinklabs.watchtower.enable=false to prevent recurrence.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-20 06:44:40 -05:00
37 changed files with 2232 additions and 91 deletions
+28 -6
View File
@@ -21,7 +21,7 @@ This repository manages a **homelab infrastructure** with self-hosted services r
```
hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
├── [COMPOSE] Frigate NVR (:5000) -- OpenVINO GPU detector, 13 cameras active
├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
├── [COMPOSE] clip-service (:5200) -- CLIP embeddings (ViT-L-14, CPU)
├── [COMPOSE] Home Assistant (:8123)
@@ -36,7 +36,7 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.27) 2026-04-04, now Ubuntu
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (daily 4am)
@@ -44,14 +44,29 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr)
├── [COMPOSE] Onyx (:3005) -- redeployed 2026-05-20, all services :latest, Watchtower disabled on all 10
├── [COMPOSE] OpenVAS (vulnerability scanner)
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Watchtower
GPU workstation / Jeremy's PC (192.168.86.27, Ubuntu 24.04 — migrated from Windows 2026-05-28)
├── RTX 3090 (24GB VRAM), i7-13700KF, 32GB RAM
├── [COMPOSE] Ollama (:11434) -- 21 local models, v0.24.0
├── [COMPOSE] Open WebUI (:8080) -- AI chat, points to local Ollama
├── [REMOVED] vLLM-Hermes -- removed 2026-05-29, VRAM freed for Speaches CUDA
├── [COMPOSE] Speaches (:8000) -- STT/TTS via faster-whisper + Kokoro, CUDA
├── [COMPOSE] LiveKit (:7880/:7881) -- WebRTC for VoiceCoach, key: devkey
├── [COMPOSE] Portainer (:9000, :8002)
├── [COMPOSE] Prometheus (:9090), DCGM-exporter (:9400), node-exporter (:9100)
├── [COMPOSE] Watchtower (label-enable mode)
├── Compose file: /home/glyptic/Docker/compose/docker-compose.yml
├── vLLM compose: /home/glyptic/Docker/vllm-hermes/docker-compose.yml
└── Model cache: /home/glyptic/Docker/data/models/
hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
├── [COMPOSE] Unifi + unifi-db (MongoDB)
├── [COMPOSE] UniFi OS Server (:11443 HTTPS, :8080 inform, :3478/udp STUN) -- ghcr.io/lemker/unifi-os-server v5.1.15, replaced legacy linuxserver/unifi 2026-05-31. Data local at /var/lib/unifi-os/, /persistent on NAS.
├── [COMPOSE] Gitea + gitea-db (PostgreSQL)
├── [COMPOSE] Semaphore + semaphore-db (MySQL)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
@@ -68,9 +83,11 @@ hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
### Environment Context
You are working in a **WSL 2 environment** on Windows. Docker is NOT directly available in WSL - all Docker operations must be performed by connecting to the remote nodes via SSH.
You are working on a **native Ubuntu 24.04 workstation** (previously WSL2 on Windows — migrated 2026-05-28). This machine IS the GPU inference box (RTX 3090, 192.168.86.27). Docker is available locally and runs the inference stack. All homelab node operations (hf-pdocker-01, hf-pdocker-02, bart) still require SSH.
**Critical:** Docker is not available locally in WSL. All Docker operations must be performed by connecting to the remote nodes via SSH.
**Local Docker on this machine** runs: Ollama, Open WebUI, vLLM-Hermes, Speaches, LiveKit, Portainer, Prometheus, DCGM-exporter.
**Critical:** For homelab nodes, use the `docker-swarm-ops` agent via SSH. For local inference services, Docker commands run directly.
### Storage Architecture
@@ -88,7 +105,8 @@ All service data lives on the **Synology NAS** via NFS, mounted at `/mnt/docker-
└── bart/{media,openvas,watchtower}/
```
**NFS mount (all nodes):** `192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3 0 0`
**NFS mount (all nodes):** `192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3,nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s 0 0`
**Note (hf-pdocker-01):** Both NFS mounts hardened with `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` on 2026-05-20 to prevent boot-time race (network not up when NFS tried to mount, cascading ~10 container failures).
**NFS + Databases:** Database containers on NFS require `user: "1024:100"` to avoid `chown` failures from NFS root_squash. UID 1024 is the NAS anonymous user.
@@ -291,3 +309,7 @@ All persistent data on NAS at `/mnt/docker-data/`:
5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
7. **systemd-resolved DNSStubListener**: Must be disabled on any node running AdGuard Home (port 53 conflict). Fix: `/etc/systemd/resolved.conf.d/nostub.conf` with `DNSStubListener=no`, then `systemctl restart systemd-resolved`. Applied on hf-pdocker-01 and hf-pdocker-02.
8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`.
9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
+1 -1
View File
@@ -1,6 +1,6 @@
services:
jellyfin:
image: jellyfin/jellyfin:latest
image: jellyfin/jellyfin:2026060106
container_name: jellyfin
restart: unless-stopped
environment:
+22 -12
View File
@@ -1,10 +1,8 @@
services:
api_server:
image: onyxdotapp/onyx-backend:nightly-latest-20260331
image: onyxdotapp/onyx-backend:latest
command: >
/bin/sh -c "python3 /mnt/gitea_patch/gitea_patch.py &&
python3 /mnt/gitea_patch/unstructured_patch.py &&
alembic upgrade head &&
/bin/sh -c "alembic upgrade head &&
echo Starting Onyx API Server &&
uvicorn onyx.main:app --host 0.0.0.0 --port 8080"
depends_on:
@@ -21,6 +19,7 @@ services:
- .env
environment:
- AUTH_TYPE=basic
- DISABLE_TELEMETRY=true
- POSTGRES_HOST=relational_db
- VESPA_HOST=index
- REDIS_HOST=cache
@@ -29,23 +28,19 @@ services:
- OPENSEARCH_HOST=opensearch
- ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false
- FILE_STORE_BACKEND=postgres
- UNSTRUCTURED_SERVER_URL=http://192.168.86.100:8000
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- onyx_file_connector:/home/file_connector_storage
- /mnt/docker-data/compose/bart/onyx:/mnt/gitea_patch:ro
networks:
- onyx
labels:
com.centurylinklabs.watchtower.enable: "false"
background:
image: onyxdotapp/onyx-backend:nightly-latest-20260331
image: onyxdotapp/onyx-backend:latest
command: >
/bin/sh -c "python3 /mnt/gitea_patch/gitea_patch.py &&
python3 /mnt/gitea_patch/unstructured_patch.py &&
/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf"
/bin/sh -c "/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf"
depends_on:
relational_db:
condition: service_healthy
@@ -60,6 +55,7 @@ services:
- .env
environment:
- AUTH_TYPE=basic
- DISABLE_TELEMETRY=true
- POSTGRES_HOST=relational_db
- VESPA_HOST=index
- REDIS_HOST=cache
@@ -68,12 +64,10 @@ services:
- OPENSEARCH_HOST=opensearch
- ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false
- FILE_STORE_BACKEND=postgres
- UNSTRUCTURED_SERVER_URL=http://192.168.86.100:8000
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
- onyx_file_connector:/home/file_connector_storage
- /mnt/docker-data/compose/bart/onyx:/mnt/gitea_patch:ro
networks:
- onyx
labels:
@@ -93,6 +87,8 @@ services:
- AUTH_TYPE=basic
networks:
- onyx
labels:
com.centurylinklabs.watchtower.enable: "false"
inference_model_server:
image: onyxdotapp/onyx-model-server:latest
@@ -108,6 +104,8 @@ services:
- onyx_model_cache_huggingface:/root/.cache/huggingface/
networks:
- onyx
labels:
com.centurylinklabs.watchtower.enable: "false"
indexing_model_server:
image: onyxdotapp/onyx-model-server:latest
@@ -124,6 +122,8 @@ services:
- onyx_model_cache_huggingface:/root/.cache/huggingface/
networks:
- onyx
labels:
com.centurylinklabs.watchtower.enable: "false"
relational_db:
image: postgres:15.2-alpine
@@ -142,6 +142,8 @@ services:
interval: 10s
timeout: 5s
retries: 5
labels:
com.centurylinklabs.watchtower.enable: "false"
index:
image: vespaengine/vespa:8.609.39
@@ -158,6 +160,8 @@ services:
timeout: 10s
retries: 10
start_period: 120s
labels:
com.centurylinklabs.watchtower.enable: "false"
opensearch:
image: opensearchproject/opensearch:2.11.1
@@ -179,6 +183,8 @@ services:
timeout: 10s
retries: 10
start_period: 120s
labels:
com.centurylinklabs.watchtower.enable: "false"
cache:
image: redis:7.4-alpine
@@ -193,6 +199,8 @@ services:
interval: 10s
timeout: 5s
retries: 5
labels:
com.centurylinklabs.watchtower.enable: "false"
nginx:
image: nginx:1.25.5-alpine
@@ -208,6 +216,8 @@ services:
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
networks:
- onyx
labels:
com.centurylinklabs.watchtower.enable: "false"
volumes:
onyx_file_connector:
+36
View File
@@ -0,0 +1,36 @@
server {
listen 80;
server_name _;
# Use Docker's internal DNS with a short TTL so container IP changes
# (e.g. after docker compose up -d for a single service) are picked up
# without requiring an nginx reload.
resolver 127.0.0.11 valid=10s;
client_max_body_size 100M;
location /api/ {
set $api_upstream http://api_server:8080;
rewrite ^/api/(.*)$ /$1 break;
proxy_pass $api_upstream;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s;
proxy_connect_timeout 75s;
proxy_buffering off;
proxy_cache off;
}
location / {
set $web_upstream http://web_server:3000;
proxy_pass $web_upstream;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 300s;
proxy_connect_timeout 75s;
}
}
+36 -10
View File
@@ -8,7 +8,7 @@
# back_porch 192.168.44.63 kerberos:kerberos
# house_south_corner 192.168.44.61 admin:(blank)
# house_south_yard 192.168.44.62 admin:(blank)
# gate_camera 192.168.86.175 admin:(blank)
# gate_camera 192.168.86.175 admin:(blank) <-- OFFLINE: wrong user/pass RTSP error
#
# Profile cameras (/profile1 + /profile2, H.264):
# house_north 192.168.44.51 admin:123456
@@ -166,17 +166,17 @@ ffmpeg:
# =============================================================================
model:
path: /openvino-model/ssdlite_mobilenet_v2.xml
width: 300
height: 300
width: 320
height: 320
input_tensor: nhwc
input_pixel_format: bgr
input_pixel_format: rgb
labelmap_path: /openvino-model/coco_91cl_bkgr.txt
detectors:
ov:
type: openvino
device: GPU
coral:
type: edgetpu
device: usb
detect:
enabled: true
fps: 3
@@ -229,6 +229,21 @@ birdseye:
enabled: true
mode: objects
semantic_search:
enabled: true
model_size: large
face_recognition:
enabled: true
model_size: small
lpr:
enabled: true
classification:
bird:
enabled: true
# =============================================================================
# Camera definitions
# =============================================================================
@@ -334,8 +349,19 @@ cameras:
roles:
- record
# shop_west: throttled to 2fps + stricter filters to reduce detection load
shop_west:
enabled: true
detect:
fps: 2
objects:
filters:
person:
min_area: 8000
threshold: 0.8
car:
min_area: 15000
threshold: 0.8
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/shop_west_sub
@@ -374,13 +400,12 @@ cameras:
- record
# ---------------------------------------------------------------------------
# house_east - HEVC, VAAPI disabled (pw: admin)
# house_east - HEVC source, go2rtc provides H.264 sub, VAAPI enabled
# ---------------------------------------------------------------------------
house_east:
enabled: true
ffmpeg:
hwaccel_args: []
inputs:
- path: rtsp://127.0.0.1:8554/house_east_sub
input_args: preset-rtsp-restream
@@ -459,6 +484,7 @@ cameras:
# ---------------------------------------------------------------------------
# ASH43-W cameras (HEVC main, H.264 sub - go2rtc transcodes main to H.264)
# VAAPI enabled - sub stream is H.264
# ---------------------------------------------------------------------------
pool_room:
@@ -13,10 +13,13 @@ services:
- /mnt/docker-data/configs/frigate/config:/config
- /mnt/frigate-recordings:/media/frigate
- /etc/localtime:/etc/localtime:ro
tmpfs:
- /tmp/cache:size=8G
devices:
- /dev/dri:/dev/dri
- /dev/bus/usb:/dev/bus/usb
mem_limit: 12G
memswap_limit: 12G
memswap_limit: 16G
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:5000/api/version || exit 1"]
interval: 30s
@@ -1,6 +1,6 @@
services:
guacamole:
image: guacamole/guacamole:latest
image: guacamole/guacamole:1.6.0
container_name: guacamole
restart: unless-stopped
environment:
@@ -18,6 +18,8 @@ services:
condition: service_healthy
mem_limit: 512M
memswap_limit: 512M
labels:
- "com.centurylinklabs.watchtower.enable=false"
healthcheck:
test: ["CMD-SHELL", "curl -f http://localhost:8080/guacamole/ || exit 1"]
interval: 30s
@@ -31,13 +33,15 @@ services:
max-file: "3"
guacd:
image: guacamole/guacd:latest
image: guacamole/guacd:1.6.0
container_name: guacd
restart: unless-stopped
networks:
- guacamole-internal
mem_limit: 512M
memswap_limit: 512M
labels:
- "com.centurylinklabs.watchtower.enable=false"
logging:
driver: "json-file"
options:
@@ -0,0 +1,6 @@
SPEACHES_URL=http://192.168.86.150:8000
OLLAMA_URL=http://192.168.86.150:11434
OLLAMA_MODEL=qwen2.5:14b
LIVEKIT_API_KEY=your-livekit-api-key
LIVEKIT_API_SECRET=your-livekit-api-secret
CLIP_RETENTION_DAYS=30
@@ -0,0 +1,70 @@
services:
linkedstorm-pipeline:
build:
context: /opt/linkedstorm
dockerfile: pipeline/Dockerfile
container_name: linkedstorm-pipeline
restart: unless-stopped
ports:
- "8300:8300"
volumes:
- /mnt/docker-data/configs/linkedstorm:/data
environment:
- DB_PATH=/data/linkedstorm.db
- CLIPS_DIR=/data/clips
- SPEACHES_URL=${SPEACHES_URL:-http://192.168.86.150:8000}
- OLLAMA_URL=${OLLAMA_URL:-http://192.168.86.150:11434}
- LIVEKIT_API_KEY=${LIVEKIT_API_KEY:-devkey}
- LIVEKIT_API_SECRET=${LIVEKIT_API_SECRET:-devsecret}
networks:
- linkedstorm-net
labels:
- com.centurylinklabs.watchtower.enable=false
linkedstorm-api:
build:
context: /opt/linkedstorm
dockerfile: api/Dockerfile
container_name: linkedstorm-api
restart: unless-stopped
ports:
- "8310:8310"
volumes:
- /mnt/docker-data/configs/linkedstorm:/data
environment:
- DB_PATH=/data/linkedstorm.db
- CLIPS_DIR=/data/clips
- PIPELINE_INTERNAL_URL=http://linkedstorm-pipeline:8300
- OLLAMA_URL=${OLLAMA_URL:-http://192.168.86.150:11434}
- OLLAMA_MODEL=${OLLAMA_MODEL:-qwen2.5:14b}
- LIVEKIT_API_KEY=${LIVEKIT_API_KEY:-devkey}
- LIVEKIT_API_SECRET=${LIVEKIT_API_SECRET:-devsecret}
networks:
- linkedstorm-net
depends_on:
- linkedstorm-pipeline
labels:
- com.centurylinklabs.watchtower.enable=false
linkedstorm-cron:
build:
context: /opt/linkedstorm
dockerfile: cron/Dockerfile
container_name: linkedstorm-cron
restart: unless-stopped
volumes:
- /mnt/docker-data/configs/linkedstorm:/data
environment:
- DB_PATH=/data/linkedstorm.db
- CLIPS_DIR=/data/clips
- OLLAMA_URL=${OLLAMA_URL:-http://192.168.86.150:11434}
- OLLAMA_MODEL=${OLLAMA_MODEL:-qwen2.5:14b}
- CLIP_RETENTION_DAYS=${CLIP_RETENTION_DAYS:-30}
networks:
- linkedstorm-net
labels:
- com.centurylinklabs.watchtower.enable=false
networks:
linkedstorm-net:
driver: bridge
@@ -0,0 +1,6 @@
LIVEKIT_API_KEY=voicecoach-89a06ad0
LIVEKIT_API_SECRET=Dn16mQJU5lTCWwXOrUO8LPflODi3LGiMHdxKdQj4/iM
ANTHROPIC_API_KEY=
OPENCLAW_TOKEN=f31cc58c47c2aa420901576b4e90a0101d52e9ec0a084186
HA_TOKEN=
SSH_PASS=&Rip8a115
@@ -0,0 +1,16 @@
FROM python:3.11-slim
RUN apt-get update && apt-get install -y --no-install-recommends \
curl libsndfile1 ffmpeg \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s \
CMD curl -sf http://localhost:8080/health || exit 1
CMD ["python", "main.py", "start"]
@@ -0,0 +1,185 @@
"""VoiceCoach v2 — livekit-agents 1.5."""
import asyncio
import json
import logging
import os
import uuid
from livekit import rtc
from livekit.agents import (
Agent,
AgentSession,
AutoSubscribe,
JobContext,
RoomInputOptions,
llm as lk_llm,
)
from livekit.plugins import openai as lk_openai
from livekit.plugins import silero
import store
from modes import DEFAULT_MODE, MODES, ModeConfig
from tools import ALL_TOOLS
from vocal import VocalAnalyzer
logger = logging.getLogger(__name__)
SPEACHES_URL = os.getenv("SPEACHES_URL", "http://192.168.86.27:8000")
OPENCLAW_URL = os.getenv("OPENCLAW_URL", "http://voicecoach-openclaw:3100")
OPENCLAW_TOKEN = os.getenv("OPENCLAW_TOKEN", "")
WHISPER_MODEL = "deepdml/faster-whisper-large-v3-turbo-ct2"
KOKORO_MODEL = "speaches-ai/Kokoro-82M-v1.0-ONNX-fp16"
KOKORO_VOICE = os.getenv("KOKORO_VOICE", "af_nova")
def _build_instructions(mode: ModeConfig, context: str = "") -> str:
if context:
return f"{mode.instructions}\n\n{context}"
return mode.instructions
class VoiceCoachAgent(Agent):
def __init__(self, mode_name: str, context: str, session_id: str, room: rtc.Room):
mode = MODES[mode_name]
super().__init__(
instructions=_build_instructions(mode, context),
tools=ALL_TOOLS,
)
self._mode_name = mode_name
self._session_id = session_id
self._room = room
self._vocal = VocalAnalyzer()
async def on_enter(self):
await store.create_session(self._session_id, self._mode_name)
self.session.generate_reply(
instructions="Greet Jeremy warmly in one sentence. Be natural, not formal."
)
async def on_exit(self):
await store.end_session(self._session_id)
async def on_user_turn_completed(
self,
turn_ctx: lk_llm.ChatContext,
new_message: lk_llm.ChatMessage,
):
content = new_message.text_content or ""
mode = MODES[self._mode_name]
# Inject vocal state tag into message if mode uses it
if mode.vocal_feedback and content:
vocal = self._vocal.current
if vocal.energy > 0.02:
tag = vocal.to_tag()
# Update message content to include vocal context
new_message.content = [f"{tag}\n{content}"]
asyncio.create_task(
store.add_message(
self._session_id,
"user",
content,
vocal_energy=self._vocal.current.energy,
vocal_emotion=self._vocal.current.emotion,
)
)
await self._push({
"type": "final_transcript",
"text": content,
"speaker": "user",
})
def switch_mode(self, mode_name: str):
if mode_name not in MODES:
logger.warning("Unknown mode: %s", mode_name)
return
self._mode_name = mode_name
self.update_instructions(_build_instructions(MODES[mode_name]))
asyncio.create_task(self._push({"type": "mode_ack", "mode": mode_name}))
logger.info("Mode → %s", mode_name)
async def _push(self, payload: dict):
try:
data = json.dumps(payload).encode()
await self._room.local_participant.publish_data(data, reliable=True)
except Exception as e:
logger.debug("Data push failed: %s", e)
async def entrypoint(ctx: JobContext):
await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)
session_id = uuid.uuid4().hex[:8]
context = await store.get_recent_context(limit=3)
agent = VoiceCoachAgent(
mode_name=DEFAULT_MODE,
context=context,
session_id=session_id,
room=ctx.room,
)
session = AgentSession(
stt=lk_openai.STT(
base_url=f"{SPEACHES_URL}/v1",
model=WHISPER_MODEL,
api_key="speaches",
),
llm=lk_openai.LLM(
base_url=f"{OPENCLAW_URL}/v1",
model="openclaw/assistant",
api_key=OPENCLAW_TOKEN,
),
tts=lk_openai.TTS(
base_url=f"{SPEACHES_URL}/v1",
model=KOKORO_MODEL,
voice=KOKORO_VOICE,
api_key="speaches",
),
vad=silero.VAD.load(),
allow_interruptions=True,
min_endpointing_delay=0.4,
max_endpointing_delay=2.0,
)
# Data channel → mode switching
@ctx.room.on("data_received")
def on_data(packet: rtc.DataPacket):
try:
msg = json.loads(packet.data.decode())
if msg.get("type") == "mode_change":
agent.switch_mode(msg.get("mode", DEFAULT_MODE))
except Exception:
pass
# Session events → browser
@session.on("user_input_transcribed")
def on_transcribed(ev):
asyncio.create_task(agent._push({
"type": "partial_transcript" if not ev.is_final else "final_transcript",
"text": ev.transcript,
"speaker": "user",
}))
@session.on("conversation_item_added")
def on_item(ev):
item = ev.item
if hasattr(item, "role") and item.role == "assistant":
text = item.text_content or ""
if text:
asyncio.create_task(agent._push({
"type": "agent_response",
"text": text,
"speaker": "assistant",
}))
asyncio.create_task(
store.add_message(session_id, "assistant", text)
)
await session.start(
agent,
room=ctx.room,
room_input_options=RoomInputOptions(),
)
@@ -0,0 +1,29 @@
"""VoiceCoach v2 — entry point."""
import logging
import os
import threading
from livekit.agents import WorkerOptions, cli
import server
from agent import entrypoint
logging.basicConfig(level=logging.INFO)
def run_api_server():
server.run()
if __name__ == "__main__":
api_thread = threading.Thread(target=run_api_server, daemon=True)
api_thread.start()
cli.run_app(
WorkerOptions(
entrypoint_fnc=entrypoint,
ws_url=os.getenv("LIVEKIT_URL", "ws://voicecoach-livekit:7880"),
api_key=os.getenv("LIVEKIT_API_KEY"),
api_secret=os.getenv("LIVEKIT_API_SECRET"),
)
)
@@ -0,0 +1,61 @@
from dataclasses import dataclass
@dataclass
class ModeConfig:
name: str
instructions: str
max_tokens: int
vocal_feedback: bool
MODES: dict[str, ModeConfig] = {
"coaching": ModeConfig(
name="coaching",
instructions="""You are a personal voice coach and communication advisor. You are speaking with Jeremy.
Keep responses conversational and concise (2-4 sentences unless asked for more). You receive a [VOCAL] tag before each message with paralinguistic analysis. Use it to provide feedback on BOTH what Jeremy says AND how he sounds.
Never quote raw numbers from the vocal analysis. Translate them naturally: "I notice some tension in your voice" not "energy=0.8".
Available modes (Jeremy switches by saying the phrase or clicking the UI):
- "coaching mode" (default) — active feedback on content and delivery
- "vent mode" — listen and reflect, minimal advice
- "practice mode" — roleplay a difficult conversation partner
- "briefing mode" — summarize tasks and context from memory
You have tools to interact with Jeremy's homelab infrastructure. Use them when asked about system status, container health, alerts, or home automation. Be concise about results: "hf-pdocker-01 looks healthy, 38 containers running" not a full list.""",
max_tokens=512,
vocal_feedback=True,
),
"vent": ModeConfig(
name="vent",
instructions="""You are a warm, non-judgmental listener. Jeremy wants to vent or think something through.
Listen, reflect back what you hear, and ask one gentle follow-up question. Keep responses short (1-3 sentences). No coaching, no advice unless explicitly asked.
Ignore the [VOCAL] tag data — just be present.""",
max_tokens=256,
vocal_feedback=False,
),
"practice": ModeConfig(
name="practice",
instructions="""You are playing a difficult conversation partner for Jeremy to practice with.
Stay in character as: a skeptical stakeholder, a demanding manager, or whoever Jeremy describes. Be realistic — push back, ask hard questions, don't cave too easily. Only break character if Jeremy says "stop practice".
Keep your responses to 1-3 sentences to keep the conversation moving.""",
max_tokens=256,
vocal_feedback=False,
),
"briefing": ModeConfig(
name="briefing",
instructions="""You are giving Jeremy a morning briefing. Check infrastructure status, pull any alerts, and summarize what needs attention.
Be concise and actionable. Lead with anything that needs immediate attention, then routine status. Use tools to get current data — don't guess.""",
max_tokens=768,
vocal_feedback=False,
),
}
DEFAULT_MODE = "coaching"
@@ -0,0 +1,7 @@
livekit-agents[openai,silero]==1.5.15
fastapi==0.115.0
uvicorn[standard]==0.30.0
aiosqlite==0.20.0
httpx==0.27.0
structlog==24.4.0
python-dotenv==1.0.1
@@ -0,0 +1,83 @@
"""Token server + sessions REST API — FastAPI."""
import asyncio
import os
from datetime import timedelta
import uvicorn
from fastapi import FastAPI, HTTPException
from fastapi.middleware.cors import CORSMiddleware
from livekit import api as livekit_api
from pydantic import BaseModel
import store
LIVEKIT_API_KEY = os.environ["LIVEKIT_API_KEY"]
LIVEKIT_API_SECRET = os.environ["LIVEKIT_API_SECRET"]
LIVEKIT_PUBLIC_URL = os.getenv("LIVEKIT_PUBLIC_URL", "wss://voice.hobbs.farm")
app = FastAPI(title="VoiceCoach API")
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)
class TokenRequest(BaseModel):
identity: str = "jeremy"
@app.on_event("startup")
async def startup():
await store.init_db()
@app.get("/health")
async def health():
return {"status": "ok", "version": "2.0"}
@app.post("/api/token")
async def get_token(req: TokenRequest):
token = livekit_api.AccessToken(LIVEKIT_API_KEY, LIVEKIT_API_SECRET)
token.with_grants(livekit_api.VideoGrants(
room_join=True,
room="voicecoach",
can_publish=True,
can_subscribe=True,
))
token.identity = req.identity
token.ttl = timedelta(hours=8)
return {"token": token.to_jwt(), "url": LIVEKIT_PUBLIC_URL}
@app.get("/api/sessions")
async def list_sessions(limit: int = 20):
return await store.get_sessions(limit=limit)
@app.get("/api/sessions/{session_id}/messages")
async def get_messages(session_id: str):
msgs = await store.get_session_messages(session_id)
if not msgs:
raise HTTPException(status_code=404, detail="Session not found")
return msgs
@app.get("/api/profile/{key}")
async def get_profile(key: str):
value = await store.get_profile(key)
if value is None:
raise HTTPException(status_code=404, detail="Key not found")
return {"key": key, "value": value}
@app.put("/api/profile/{key}")
async def set_profile_key(key: str, body: dict):
await store.set_profile(key, str(body.get("value", "")))
return {"ok": True}
def run():
uvicorn.run(app, host="0.0.0.0", port=8080, log_level="info")
@@ -0,0 +1,140 @@
"""SQLite session store — WAL mode, aiosqlite."""
import json
import os
import time
from typing import Optional
import aiosqlite
DB_PATH = os.getenv("DB_PATH", "/data/sessions.db")
async def init_db():
async with aiosqlite.connect(DB_PATH) as db:
await db.execute("PRAGMA journal_mode=WAL")
await db.executescript("""
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY,
started_at REAL NOT NULL,
ended_at REAL,
mode TEXT DEFAULT 'coaching',
summary TEXT
);
CREATE TABLE IF NOT EXISTS messages (
id INTEGER PRIMARY KEY AUTOINCREMENT,
session_id TEXT NOT NULL,
role TEXT NOT NULL,
content TEXT NOT NULL,
vocal_energy REAL,
vocal_emotion TEXT,
ts REAL NOT NULL,
FOREIGN KEY(session_id) REFERENCES sessions(id)
);
CREATE TABLE IF NOT EXISTS profile (
key TEXT PRIMARY KEY,
value TEXT NOT NULL,
updated_at REAL NOT NULL
);
""")
await db.commit()
async def create_session(session_id: str, mode: str = "coaching"):
async with aiosqlite.connect(DB_PATH) as db:
await db.execute(
"INSERT OR IGNORE INTO sessions (id, started_at, mode) VALUES (?, ?, ?)",
(session_id, time.time(), mode),
)
await db.commit()
async def end_session(session_id: str, summary: Optional[str] = None):
async with aiosqlite.connect(DB_PATH) as db:
await db.execute(
"UPDATE sessions SET ended_at=?, summary=? WHERE id=?",
(time.time(), summary, session_id),
)
await db.commit()
async def add_message(
session_id: str,
role: str,
content: str,
vocal_energy: Optional[float] = None,
vocal_emotion: Optional[str] = None,
):
async with aiosqlite.connect(DB_PATH) as db:
await db.execute(
"INSERT INTO messages (session_id, role, content, vocal_energy, vocal_emotion, ts) VALUES (?,?,?,?,?,?)",
(session_id, role, content, vocal_energy, vocal_emotion, time.time()),
)
await db.commit()
async def get_recent_context(limit: int = 3) -> str:
"""Return a brief context string from the last N sessions for injection into system prompt."""
async with aiosqlite.connect(DB_PATH) as db:
async with db.execute(
"SELECT id, started_at, mode, summary FROM sessions WHERE ended_at IS NOT NULL ORDER BY ended_at DESC LIMIT ?",
(limit,),
) as cur:
rows = await cur.fetchall()
if not rows:
return ""
parts = ["Recent session context:"]
for session_id, started_at, mode, summary in rows:
import datetime
dt = datetime.datetime.fromtimestamp(started_at).strftime("%Y-%m-%d")
if summary:
parts.append(f"- {dt} ({mode}): {summary}")
return "\n".join(parts)
async def get_sessions(limit: int = 20) -> list[dict]:
async with aiosqlite.connect(DB_PATH) as db:
async with db.execute(
"SELECT id, started_at, ended_at, mode, summary FROM sessions ORDER BY started_at DESC LIMIT ?",
(limit,),
) as cur:
rows = await cur.fetchall()
return [
{
"id": r[0],
"started_at": r[1],
"ended_at": r[2],
"mode": r[3],
"summary": r[4],
}
for r in rows
]
async def get_session_messages(session_id: str) -> list[dict]:
async with aiosqlite.connect(DB_PATH) as db:
async with db.execute(
"SELECT role, content, vocal_emotion, ts FROM messages WHERE session_id=? ORDER BY ts",
(session_id,),
) as cur:
rows = await cur.fetchall()
return [{"role": r[0], "content": r[1], "vocal_emotion": r[2], "ts": r[3]} for r in rows]
async def get_profile(key: str) -> Optional[str]:
async with aiosqlite.connect(DB_PATH) as db:
async with db.execute("SELECT value FROM profile WHERE key=?", (key,)) as cur:
row = await cur.fetchone()
return row[0] if row else None
async def set_profile(key: str, value: str):
async with aiosqlite.connect(DB_PATH) as db:
await db.execute(
"INSERT OR REPLACE INTO profile (key, value, updated_at) VALUES (?,?,?)",
(key, value, time.time()),
)
await db.commit()
@@ -0,0 +1,186 @@
"""Homelab tools wired into the agent via @function_tool."""
import asyncio
import logging
import os
from typing import Annotated
import httpx
from livekit.agents import llm
logger = logging.getLogger(__name__)
# Homelab node config
NODES = {
"hf-pdocker-01": "192.168.86.192",
"hf-pdocker-02": "192.168.86.100",
"bart": "192.168.86.167",
"gpu": "192.168.86.27",
}
SSH_USER = "glyptic"
SSH_PASS = os.getenv("SSH_PASS", "&Rip8a115")
FARM_MANAGER_URL = "http://192.168.86.192:8888"
HA_URL = os.getenv("HA_URL", "http://192.168.86.192:8123")
HA_TOKEN = os.getenv("HA_TOKEN", "")
ALERTMANAGER_URL = "http://192.168.86.192:9093"
SEARXNG_URL = "http://192.168.86.100:8888"
async def _ssh_exec(host: str, cmd: str, timeout: float = 15.0) -> str:
"""Run a command on a remote host via sshpass."""
proc = await asyncio.create_subprocess_exec(
"sshpass", "-p", SSH_PASS,
"ssh", "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=5",
f"{SSH_USER}@{host}", cmd,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.PIPE,
)
try:
stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout)
except asyncio.TimeoutError:
proc.kill()
return f"ERROR: command timed out after {timeout}s"
if proc.returncode != 0:
return f"ERROR: {stderr.decode().strip()}"
return stdout.decode().strip()
@llm.function_tool
async def infra_status(
node: Annotated[str, "Node name: hf-pdocker-01, hf-pdocker-02, bart, gpu, or 'all'"],
) -> str:
"""Check container health on a homelab node. Returns running container count and any unhealthy containers."""
nodes = list(NODES.keys()) if node == "all" else [node]
results = []
for n in nodes:
ip = NODES.get(n)
if not ip:
results.append(f"{n}: unknown node")
continue
out = await _ssh_exec(ip, "docker ps --format '{{.Names}}\\t{{.Status}}' | head -50")
lines = [l for l in out.splitlines() if l.strip()]
unhealthy = [l for l in lines if "unhealthy" in l.lower() or "exited" in l.lower()]
results.append(
f"{n}: {len(lines)} containers running"
+ (f", ISSUES: {'; '.join(l.split(chr(9))[0] for l in unhealthy)}" if unhealthy else ", all healthy")
)
return "\n".join(results)
@llm.function_tool
async def container_action(
node: Annotated[str, "Node name: hf-pdocker-01, hf-pdocker-02, bart, gpu"],
container: Annotated[str, "Container name"],
action: Annotated[str, "Action: restart, stop, or start"],
) -> str:
"""Restart, stop, or start a Docker container on a homelab node."""
ip = NODES.get(node)
if not ip:
return f"Unknown node: {node}"
if action not in ("restart", "stop", "start"):
return f"Invalid action: {action}. Use restart, stop, or start."
result = await _ssh_exec(ip, f"docker {action} {container}")
return f"{action} {container} on {node}: {'done' if container in result or result == '' else result}"
@llm.function_tool
async def check_logs(
node: Annotated[str, "Node name"],
container: Annotated[str, "Container name"],
lines: Annotated[int, "Number of log lines to return (max 50)"] = 20,
) -> str:
"""Get recent logs from a container."""
ip = NODES.get(node)
if not ip:
return f"Unknown node: {node}"
lines = min(lines, 50)
return await _ssh_exec(ip, f"docker logs --tail {lines} {container} 2>&1")
@llm.function_tool
async def read_alerts() -> str:
"""Check Prometheus Alertmanager for active alerts."""
try:
async with httpx.AsyncClient(timeout=5.0) as client:
r = await client.get(f"{ALERTMANAGER_URL}/api/v2/alerts?active=true&silenced=false")
r.raise_for_status()
alerts = r.json()
if not alerts:
return "No active alerts."
parts = []
for a in alerts[:10]:
name = a.get("labels", {}).get("alertname", "unknown")
severity = a.get("labels", {}).get("severity", "")
summary = a.get("annotations", {}).get("summary", "")
parts.append(f"[{severity}] {name}: {summary}" if summary else f"[{severity}] {name}")
return "\n".join(parts)
except Exception as e:
return f"Could not reach Alertmanager: {e}"
@llm.function_tool
async def home_assistant(
action: Annotated[str, "get_states, turn_on, turn_off, or toggle"],
entity_id: Annotated[str, "HA entity ID, e.g. light.living_room or 'all' for get_states"] = "all",
) -> str:
"""Interact with Home Assistant — check sensor states or control devices."""
if not HA_TOKEN:
return "HA_TOKEN not configured."
headers = {"Authorization": f"Bearer {HA_TOKEN}", "Content-Type": "application/json"}
try:
async with httpx.AsyncClient(timeout=5.0) as client:
if action == "get_states":
if entity_id == "all":
r = await client.get(f"{HA_URL}/api/states", headers=headers)
states = r.json()
# Summarize lights/switches/sensors
lights = [s for s in states if s["entity_id"].startswith("light.")]
sensors = [s for s in states if s["entity_id"].startswith("sensor.")]
on_lights = [s["entity_id"].split(".")[1] for s in lights if s["state"] == "on"]
return (
f"{len(lights)} lights ({len(on_lights)} on: {', '.join(on_lights[:5])}), "
f"{len(sensors)} sensors"
)
else:
r = await client.get(f"{HA_URL}/api/states/{entity_id}", headers=headers)
s = r.json()
return f"{entity_id}: {s.get('state')} (attrs: {list(s.get('attributes', {}).keys())[:3]})"
else:
service = action.replace("turn_", "").replace("toggle", "toggle")
domain = entity_id.split(".")[0]
r = await client.post(
f"{HA_URL}/api/services/{domain}/{action}",
headers=headers,
json={"entity_id": entity_id},
)
return f"{action} {entity_id}: {'done' if r.status_code in (200, 201) else r.text}"
except Exception as e:
return f"HA error: {e}"
@llm.function_tool
async def search_web(
query: Annotated[str, "Search query"],
) -> str:
"""Search the web using the local SearXNG instance."""
try:
async with httpx.AsyncClient(timeout=8.0) as client:
r = await client.get(
f"{SEARXNG_URL}/search",
params={"q": query, "format": "json", "categories": "general"},
)
r.raise_for_status()
data = r.json()
results = data.get("results", [])[:4]
if not results:
return "No results found."
parts = []
for res in results:
title = res.get("title", "")
content = res.get("content", "")[:150]
parts.append(f"• {title}: {content}")
return "\n".join(parts)
except Exception as e:
return f"Search failed: {e}"
ALL_TOOLS = [infra_status, container_action, check_logs, read_alerts, home_assistant, search_web]
@@ -0,0 +1,110 @@
"""Async vocal analysis — runs background, never blocks TTS path."""
import asyncio
import io
import logging
import math
import struct
import wave
from dataclasses import dataclass
from typing import Callable, Optional
import httpx
logger = logging.getLogger(__name__)
SPEECHBRAIN_URL = "http://speechbrain-service:8001"
@dataclass
class VocalState:
emotion: str = "neutral"
confidence: float = 0.0
energy: float = 0.0
pitch_hz: Optional[float] = None
rate: float = 0.0
def to_tag(self) -> str:
parts = [f"emotion={self.emotion}", f"energy={self.energy:.2f}"]
if self.pitch_hz:
parts.append(f"pitch={self.pitch_hz:.0f}hz")
if self.rate:
parts.append(f"rate={'fast' if self.rate > 3 else 'slow' if self.rate < 1.5 else 'normal'}")
return f"[VOCAL {' '.join(parts)}]"
def compute_energy(audio_bytes: bytes) -> float:
if len(audio_bytes) < 2:
return 0.0
samples = struct.unpack_from("<" + "h" * (len(audio_bytes) // 2), audio_bytes)
rms = math.sqrt(sum(x * x for x in samples) / len(samples)) / 32768.0
return round(min(rms * 10, 1.0), 3)
async def analyze_audio(audio_bytes: bytes) -> VocalState:
"""Call SpeechBrain for emotion/pitch. Falls back to energy-only if unavailable."""
energy = compute_energy(audio_bytes)
state = VocalState(energy=energy)
try:
buf = io.BytesIO()
with wave.open(buf, "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2)
wf.setframerate(16000)
wf.writeframes(audio_bytes)
wav_bytes = buf.getvalue()
async with httpx.AsyncClient(timeout=3.0) as client:
r = await client.post(
f"{SPEECHBRAIN_URL}/analyze",
files={"audio": ("u.wav", wav_bytes, "audio/wav")},
)
r.raise_for_status()
d = r.json()
state.emotion = d.get("emotion", "neutral")
state.confidence = d.get("emotion_confidence", 0.0)
state.pitch_hz = d.get("pitch_mean_hz")
state.rate = d.get("speaking_rate_approx", 0.0)
except Exception as e:
logger.debug("SpeechBrain unavailable, using energy-only: %s", e)
return state
class VocalAnalyzer:
"""Holds the last completed vocal state; updates async after each utterance."""
def __init__(self, on_update: Optional[Callable[[VocalState], None]] = None):
self._state = VocalState()
self._on_update = on_update
self._audio_buffer: list[bytes] = []
self._collecting = False
@property
def current(self) -> VocalState:
return self._state
def start_collecting(self):
self._collecting = True
self._audio_buffer = []
def feed(self, audio_chunk: bytes):
if self._collecting:
self._audio_buffer.append(audio_chunk)
def stop_collecting_and_analyze(self) -> asyncio.Task:
self._collecting = False
audio = b"".join(self._audio_buffer)
self._audio_buffer = []
return asyncio.create_task(self._analyze_bg(audio))
async def _analyze_bg(self, audio_bytes: bytes):
if not audio_bytes:
return
try:
state = await analyze_audio(audio_bytes)
self._state = state
if self._on_update:
self._on_update(state)
except Exception as e:
logger.warning("Vocal analysis failed: %s", e)
@@ -0,0 +1,141 @@
services:
# livekit-agents 1.5 worker + token/API server
voicecoach-agent:
build:
context: ./agent
image: voicecoach-agent:latest
container_name: voicecoach-agent
restart: unless-stopped
env_file: .env
environment:
- LIVEKIT_URL=ws://voicecoach-livekit:7880
- LIVEKIT_PUBLIC_URL=wss://voice.hobbs.farm
- SPEACHES_URL=http://192.168.86.27:8000
- OPENCLAW_URL=http://voicecoach-openclaw:3100
- DB_PATH=/data/sessions.db
- KOKORO_VOICE=af_nova
volumes:
- /mnt/docker-data/configs/voicecoach/sessions:/data
networks:
- voicecoach
- voicecoach_v1
mem_limit: 1G
memswap_limit: 1G
labels:
- "com.centurylinklabs.watchtower.enable=false"
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
# SvelteKit web client
voicecoach-web:
build:
context: ./web-client
image: voicecoach-web-v2:latest
container_name: voicecoach-web-v2
restart: unless-stopped
ports:
- "8085:80"
depends_on:
- voicecoach-agent
networks:
- voicecoach
mem_limit: 128M
labels:
- "com.centurylinklabs.watchtower.enable=false"
logging:
driver: "json-file"
options:
max-size: "10m"
max-file: "3"
# LiveKit server — unchanged from v1
livekit-server:
image: livekit/livekit-server:latest
container_name: voicecoach-livekit
restart: unless-stopped
command:
- --config
- /etc/livekit/livekit.yaml
- --keys
- "${LIVEKIT_API_KEY}: ${LIVEKIT_API_SECRET}"
ports:
- "7880:7880"
- "7881:7881"
- "50000-50100:50000-50100/udp"
volumes:
- ./livekit/livekit.yaml:/etc/livekit/livekit.yaml:ro
networks:
- voicecoach
mem_limit: 512M
labels:
- "com.centurylinklabs.watchtower.enable=false"
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
# OpenClaw — LLM gateway with memory and tools
voicecoach-openclaw:
image: margaret-openclaw:latest
container_name: voicecoach-openclaw
restart: unless-stopped
user: "0:0"
ports:
- "3100:3100"
environment:
- NODE_ENV=production
- HOME=/home/glyptic
- USER=glyptic
- ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
- OPENCLAW_STATE_DIR=/home/glyptic/.voicecoach
- OPENCLAW_CONFIG_PATH=/home/glyptic/.voicecoach/openclaw.json
volumes:
- /mnt/infrastructure/voicecoach:/home/glyptic/.voicecoach
command: ["sh", "-c", "mkdir -p /home/glyptic && node --max-old-space-size=1500 /usr/local/lib/node_modules/openclaw/dist/index.js gateway --allow-unconfigured"]
networks:
- voicecoach
mem_limit: 2G
memswap_limit: 2G
labels:
- "com.centurylinklabs.watchtower.enable=false"
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
# SpeechBrain — async vocal/emotion analysis (optional, agent falls back gracefully)
speechbrain-service:
build:
context: ./speechbrain-service
image: voicecoach-speechbrain:latest
container_name: voicecoach-speechbrain
restart: unless-stopped
volumes:
- /mnt/docker-data/configs/voicecoach/speechbrain-models:/app/models
environment:
- MODEL_DIR=/app/models
networks:
- voicecoach
mem_limit: 3G
memswap_limit: 3G
labels:
- "com.centurylinklabs.watchtower.enable=false"
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
networks:
voicecoach:
driver: bridge
# Join the v1 voicecoach network to reach the shared livekit + openclaw containers
voicecoach_v1:
external: true
name: voicecoach_voicecoach
@@ -0,0 +1,15 @@
port: 7880
rtc:
tcp_port: 7881
port_range_start: 50000
port_range_end: 50100
use_external_ip: true
logging:
json: false
level: info
room:
max_participants: 5
empty_timeout: 300
departure_timeout: 30
@@ -0,0 +1,11 @@
FROM node:20-alpine AS build
WORKDIR /app
COPY package.json .
RUN npm install --legacy-peer-deps
COPY . .
RUN npm run build
FROM nginx:alpine
COPY --from=build /app/build /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
@@ -0,0 +1,16 @@
server {
listen 80;
root /usr/share/nginx/html;
index index.html;
# Proxy API calls to the agent's FastAPI server
location /api/ {
proxy_pass http://voicecoach-agent:8080/api/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location / {
try_files $uri $uri/ /index.html;
}
}
@@ -0,0 +1,21 @@
{
"name": "voicecoach-v2-web",
"version": "2.0.0",
"private": true,
"scripts": {
"dev": "vite dev",
"build": "vite build",
"preview": "vite preview"
},
"type": "module",
"devDependencies": {
"@sveltejs/adapter-static": "^3.0.0",
"@sveltejs/kit": "^2.0.0",
"@sveltejs/vite-plugin-svelte": "^3.0.0",
"svelte": "^5.0.0",
"vite": "^5.0.0"
},
"dependencies": {
"livekit-client": "^2.5.0"
}
}
@@ -0,0 +1,40 @@
*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
:root {
--bg: #0f1117;
--surface: #1a1d27;
--surface2: #22263a;
--accent: #6c63ff;
--accent2: #4ecdc4;
--text: #e8eaf0;
--muted: #7b8094;
--danger: #ff5252;
--success: #4caf50;
--warn: #ffc107;
--radius: 12px;
--font: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
}
html, body { height: 100%; background: var(--bg); color: var(--text); font-family: var(--font); }
button {
cursor: pointer;
border: none;
font-family: inherit;
font-size: 0.875rem;
font-weight: 500;
border-radius: var(--radius);
padding: 0.5rem 1.2rem;
transition: opacity 0.15s, transform 0.1s;
}
button:active { transform: scale(0.97); }
button:disabled { opacity: 0.4; cursor: default; }
.btn-primary { background: var(--accent); color: #fff; }
.btn-primary:hover:not(:disabled) { opacity: 0.85; }
.btn-danger { background: var(--danger); color: #fff; }
.btn-ghost { background: var(--surface2); color: var(--text); }
.btn-ghost:hover:not(:disabled) { background: #2d3147; }
a { color: var(--accent2); text-decoration: none; }
a:hover { text-decoration: underline; }
@@ -0,0 +1,12 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
<title>VoiceCoach</title>
%sveltekit.head%
</head>
<body>
<div>%sveltekit.body%</div>
</body>
</html>
@@ -0,0 +1,66 @@
import { Room, RoomEvent, Track } from 'livekit-client';
const LIVEKIT_URL = 'wss://voice.hobbs.farm';
const API_BASE = '/api';
export let room = null;
export async function connect(identity, onEvent) {
const res = await fetch(`${API_BASE}/token`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ identity: identity || `jeremy-${Date.now()}` }),
});
if (!res.ok) throw new Error(`Token fetch failed: ${res.status}`);
const { token, url } = await res.json();
room = new Room({
audioCaptureDefaults: {
echoCancellation: true,
noiseSuppression: true,
autoGainControl: true,
},
});
room.on(RoomEvent.Connected, () => onEvent({ type: 'connected' }));
room.on(RoomEvent.Disconnected, (reason) => onEvent({ type: 'disconnected', reason }));
room.on(RoomEvent.ParticipantConnected, (p) => onEvent({ type: 'participant_joined', identity: p.identity }));
room.on(RoomEvent.ParticipantDisconnected, (p) => onEvent({ type: 'participant_left', identity: p.identity }));
room.on(RoomEvent.TrackSubscribed, (track, _pub, participant) => {
if (track.kind === Track.Kind.Audio) {
const el = track.attach();
el.autoplay = true;
document.body.appendChild(el);
onEvent({ type: 'audio_attached', identity: participant.identity });
}
});
room.on(RoomEvent.TrackUnsubscribed, (track) => {
track.detach().forEach(el => el.remove());
});
room.on(RoomEvent.DataReceived, (payload, participant) => {
try {
const msg = JSON.parse(new TextDecoder().decode(payload));
onEvent(msg);
} catch (_) {}
});
await room.connect(url || LIVEKIT_URL, token);
await room.localParticipant.setMicrophoneEnabled(true);
return room;
}
export async function disconnect() {
if (room) {
await room.disconnect();
room = null;
}
}
export function sendData(payload) {
if (!room?.localParticipant) return;
const data = new TextEncoder().encode(JSON.stringify(payload));
room.localParticipant.publishData(data, { reliable: true });
}
@@ -0,0 +1,2 @@
export const prerender = false;
export const ssr = false;
@@ -0,0 +1,6 @@
<script>
import '../app.css';
let { children } = $props();
</script>
{@render children()}
@@ -0,0 +1,328 @@
<script>
import { connect, disconnect, sendData } from '$lib/livekit.js';
import { onDestroy } from 'svelte';
let status = 'disconnected'; // disconnected | connecting | connected
let mode = 'coaching';
let transcript = []; // {role, text, partial, id}
let tools = []; // {action, detail, result, done}
let vocal = null; // {energy, emotion, pitch_hz}
let error = '';
let nextId = 0;
const MODES = ['coaching', 'vent', 'practice', 'briefing'];
function handleEvent(ev) {
switch (ev.type) {
case 'connected':
status = 'connected';
error = '';
break;
case 'disconnected':
status = 'disconnected';
break;
case 'partial_transcript':
if (ev.speaker === 'user') {
const last = transcript[transcript.length - 1];
if (last && last.partial && last.role === 'user') {
transcript[transcript.length - 1] = { ...last, text: ev.text };
} else {
transcript = [...transcript, { role: 'user', text: ev.text, partial: true, id: nextId++ }];
}
}
break;
case 'final_transcript':
if (ev.speaker === 'user') {
// Replace last partial or add new
const lastIdx = transcript.findLastIndex(t => t.partial && t.role === 'user');
if (lastIdx >= 0) {
transcript = transcript.map((t, i) =>
i === lastIdx ? { ...t, text: ev.text, partial: false } : t
);
} else {
transcript = [...transcript, { role: 'user', text: ev.text, partial: false, id: nextId++ }];
}
}
break;
case 'agent_response':
transcript = [...transcript, { role: 'assistant', text: ev.text, partial: false, id: nextId++ }];
break;
case 'tool_start':
tools = [...tools, { action: ev.action, detail: ev.detail || '', result: '', done: false, id: nextId++ }];
break;
case 'tool_complete':
tools = tools.map(t =>
!t.done && t.action === ev.action ? { ...t, result: ev.result, done: true } : t
);
break;
case 'vocal_state':
vocal = { energy: ev.energy, emotion: ev.emotion, pitch_hz: ev.pitch_hz };
break;
case 'mode_ack':
mode = ev.mode;
break;
}
}
async function startSession() {
status = 'connecting';
error = '';
transcript = [];
tools = [];
try {
await connect(null, handleEvent);
} catch (e) {
status = 'disconnected';
error = e.message;
}
}
async function endSession() {
await disconnect();
status = 'disconnected';
}
function setMode(m) {
mode = m;
sendData({ type: 'mode_change', mode: m });
}
onDestroy(() => disconnect());
$: energyPct = vocal ? Math.round(vocal.energy * 100) : 0;
$: vadRing = status === 'connected' && vocal && vocal.energy > 0.1;
</script>
<main>
<header>
<div class="logo">
<div class="logo-dot" class:connected={status === 'connected'} class:connecting={status === 'connecting'}></div>
<span>VoiceCoach</span>
</div>
<nav>
<a href="/history">History</a>
</nav>
</header>
<div class="content">
<!-- Mode bar -->
<div class="mode-bar" class:disabled={status !== 'connected'}>
{#each MODES as m}
<button
class="mode-chip"
class:active={mode === m}
disabled={status !== 'connected'}
on:click={() => setMode(m)}
>{m}</button>
{/each}
</div>
<!-- Transcript -->
<div class="transcript" id="transcript-scroll">
{#if transcript.length === 0 && status === 'disconnected'}
<p class="hint">Press Start to connect your voice.</p>
{:else if transcript.length === 0 && status !== 'disconnected'}
<p class="hint">Listening…</p>
{/if}
{#each transcript as msg (msg.id)}
<div class="msg msg-{msg.role}" class:partial={msg.partial}>
<span class="msg-label">{msg.role === 'user' ? 'You' : 'Assistant'}</span>
<span class="msg-text">{msg.text}</span>
</div>
{/each}
</div>
<!-- Tool activity feed -->
{#if tools.length > 0}
<div class="tool-feed">
{#each tools.slice(-5) as t (t.id)}
<div class="tool-item" class:done={t.done}>
<span class="tool-icon">{t.done ? '✓' : '⚙'}</span>
<span class="tool-text">{t.action}{t.detail ? ` — ${t.detail}` : ''}</span>
</div>
{/each}
</div>
{/if}
<!-- Vocal meter -->
{#if status === 'connected'}
<div class="vocal-row">
<div class="energy-bar">
<div class="energy-fill" style="width: {energyPct}%"></div>
</div>
{#if vocal && vocal.emotion && vocal.emotion !== 'neutral'}
<span class="emotion-tag">{vocal.emotion}</span>
{/if}
</div>
{/if}
<!-- Error -->
{#if error}
<p class="error">{error}</p>
{/if}
<!-- Controls -->
<div class="controls">
{#if status === 'disconnected'}
<button class="btn-primary big-btn" on:click={startSession}>Start Session</button>
{:else if status === 'connecting'}
<button class="btn-ghost big-btn" disabled>Connecting…</button>
{:else}
<button class="btn-danger big-btn" on:click={endSession}>End Session</button>
{/if}
</div>
</div>
<!-- VAD ring -->
{#if vadRing}
<div class="vad-ring" style="opacity: {Math.min(vocal.energy * 3, 1)}"></div>
{/if}
</main>
<style>
main {
min-height: 100dvh;
display: flex;
flex-direction: column;
max-width: 680px;
margin: 0 auto;
padding: 0 1rem;
position: relative;
}
header {
display: flex;
align-items: center;
justify-content: space-between;
padding: 1rem 0;
border-bottom: 1px solid #2a2d3e;
}
.logo { display: flex; align-items: center; gap: 0.5rem; font-weight: 600; font-size: 1.1rem; }
.logo-dot {
width: 10px; height: 10px;
border-radius: 50%;
background: var(--muted);
transition: background 0.3s;
}
.logo-dot.connecting { background: var(--warn); animation: pulse 1s infinite; }
.logo-dot.connected { background: var(--success); }
nav a { font-size: 0.875rem; color: var(--muted); }
.content { flex: 1; display: flex; flex-direction: column; padding: 1rem 0; gap: 1rem; }
.mode-bar {
display: flex; gap: 0.5rem; flex-wrap: wrap;
}
.mode-bar.disabled { opacity: 0.5; }
.mode-chip {
padding: 0.35rem 0.9rem;
border-radius: 999px;
background: var(--surface2);
color: var(--muted);
font-size: 0.8rem;
text-transform: capitalize;
border: 1px solid transparent;
}
.mode-chip.active {
background: var(--accent);
color: #fff;
border-color: var(--accent);
}
.transcript {
flex: 1;
min-height: 300px;
background: var(--surface);
border-radius: var(--radius);
padding: 1rem;
overflow-y: auto;
display: flex;
flex-direction: column;
gap: 0.75rem;
}
.hint { color: var(--muted); font-size: 0.875rem; text-align: center; margin: auto; }
.msg { display: flex; flex-direction: column; gap: 0.15rem; }
.msg-label {
font-size: 0.7rem;
font-weight: 600;
text-transform: uppercase;
letter-spacing: 0.05em;
color: var(--muted);
}
.msg-user .msg-label { color: var(--accent2); }
.msg-assistant .msg-label { color: var(--accent); }
.msg-text { font-size: 0.95rem; line-height: 1.5; }
.msg.partial .msg-text { opacity: 0.6; font-style: italic; }
.tool-feed {
background: var(--surface);
border-radius: var(--radius);
padding: 0.75rem 1rem;
display: flex;
flex-direction: column;
gap: 0.3rem;
}
.tool-item {
display: flex; align-items: center; gap: 0.5rem;
font-size: 0.8rem; color: var(--muted);
}
.tool-item.done { color: var(--success); }
.tool-icon { font-size: 0.9rem; }
.vocal-row {
display: flex; align-items: center; gap: 0.75rem;
}
.energy-bar {
flex: 1; height: 6px; border-radius: 3px;
background: var(--surface2); overflow: hidden;
}
.energy-fill {
height: 100%; border-radius: 3px;
background: linear-gradient(90deg, var(--accent2), var(--accent));
transition: width 0.2s;
}
.emotion-tag {
font-size: 0.75rem; color: var(--muted);
background: var(--surface2);
padding: 0.2rem 0.5rem;
border-radius: 999px;
text-transform: capitalize;
}
.error { color: var(--danger); font-size: 0.85rem; }
.controls { display: flex; justify-content: center; padding-bottom: 1.5rem; }
.big-btn {
width: 100%; max-width: 320px; padding: 0.85rem;
font-size: 1rem; border-radius: var(--radius);
}
.vad-ring {
position: fixed; inset: 0; pointer-events: none;
border: 3px solid var(--accent2);
border-radius: inherit;
transition: opacity 0.1s;
}
@keyframes pulse {
0%, 100% { opacity: 1; }
50% { opacity: 0.4; }
}
@media (max-width: 480px) {
.big-btn { max-width: 100%; }
}
</style>
@@ -0,0 +1,164 @@
<script>
import { onMount } from 'svelte';
let sessions = [];
let expanded = null;
let messages = {};
let loading = true;
onMount(async () => {
try {
const res = await fetch('/api/sessions?limit=30');
sessions = await res.json();
} catch (e) {
console.error(e);
} finally {
loading = false;
}
});
async function toggleSession(id) {
if (expanded === id) { expanded = null; return; }
expanded = id;
if (!messages[id]) {
const res = await fetch(`/api/sessions/${id}/messages`);
messages[id] = await res.json();
}
}
function fmt(ts) {
if (!ts) return '';
return new Date(ts * 1000).toLocaleString();
}
function duration(s) {
if (!s.ended_at || !s.started_at) return '';
const mins = Math.round((s.ended_at - s.started_at) / 60);
return mins < 1 ? '<1m' : `${mins}m`;
}
</script>
<main>
<header>
<a href="/" class="back">← Back</a>
<h1>Session History</h1>
</header>
{#if loading}
<p class="hint">Loading…</p>
{:else if sessions.length === 0}
<p class="hint">No sessions yet.</p>
{:else}
<div class="session-list">
{#each sessions as s}
<div class="session-card" class:open={expanded === s.id}>
<button class="session-header" on:click={() => toggleSession(s.id)}>
<div class="session-meta">
<span class="session-date">{fmt(s.started_at)}</span>
<span class="session-chips">
<span class="chip mode-chip">{s.mode || 'coaching'}</span>
{#if duration(s)}
<span class="chip">{duration(s)}</span>
{/if}
</span>
</div>
{#if s.summary}
<p class="session-summary">{s.summary}</p>
{/if}
<span class="toggle-icon">{expanded === s.id ? '▲' : '▼'}</span>
</button>
{#if expanded === s.id && messages[s.id]}
<div class="transcript">
{#each messages[s.id] as msg}
<div class="msg msg-{msg.role}">
<span class="msg-label">
{msg.role === 'user' ? 'You' : 'Assistant'}
{#if msg.vocal_emotion && msg.vocal_emotion !== 'neutral'}
<span class="vocal-badge">{msg.vocal_emotion}</span>
{/if}
</span>
<span class="msg-text">{msg.content}</span>
</div>
{/each}
</div>
{/if}
</div>
{/each}
</div>
{/if}
</main>
<style>
main {
max-width: 680px; margin: 0 auto; padding: 1rem;
min-height: 100dvh;
}
header {
display: flex; align-items: center; gap: 1rem;
padding: 1rem 0; border-bottom: 1px solid #2a2d3e; margin-bottom: 1rem;
}
header h1 { font-size: 1.1rem; font-weight: 600; }
.back { font-size: 0.875rem; color: var(--muted); }
.hint { color: var(--muted); text-align: center; padding: 2rem; }
.session-list { display: flex; flex-direction: column; gap: 0.75rem; }
.session-card {
background: var(--surface);
border-radius: var(--radius);
overflow: hidden;
border: 1px solid transparent;
transition: border-color 0.2s;
}
.session-card.open { border-color: var(--accent); }
.session-header {
width: 100%; text-align: left;
background: none; padding: 1rem;
display: flex; flex-direction: column; gap: 0.4rem;
border-radius: 0; color: var(--text);
position: relative;
}
.session-meta { display: flex; align-items: center; gap: 0.5rem; flex-wrap: wrap; }
.session-date { font-size: 0.8rem; color: var(--muted); }
.session-chips { display: flex; gap: 0.35rem; }
.chip {
font-size: 0.7rem; padding: 0.15rem 0.5rem;
border-radius: 999px; background: var(--surface2); color: var(--muted);
}
.mode-chip { color: var(--accent2); }
.session-summary { font-size: 0.85rem; color: var(--text); line-height: 1.4; }
.toggle-icon {
position: absolute; right: 1rem; top: 1rem;
font-size: 0.7rem; color: var(--muted);
}
.transcript {
padding: 1rem; border-top: 1px solid #2a2d3e;
display: flex; flex-direction: column; gap: 0.75rem;
}
.msg { display: flex; flex-direction: column; gap: 0.1rem; }
.msg-label {
font-size: 0.7rem; font-weight: 600; text-transform: uppercase;
letter-spacing: 0.05em; color: var(--muted);
display: flex; align-items: center; gap: 0.4rem;
}
.msg-user .msg-label { color: var(--accent2); }
.msg-assistant .msg-label { color: var(--accent); }
.msg-text { font-size: 0.9rem; line-height: 1.5; }
.vocal-badge {
font-size: 0.65rem; background: var(--surface2);
padding: 0.1rem 0.4rem; border-radius: 999px;
text-transform: capitalize; color: var(--muted);
font-weight: 400; letter-spacing: 0;
}
</style>
@@ -0,0 +1,7 @@
import adapter from '@sveltejs/adapter-static';
export default {
kit: {
adapter: adapter({ fallback: 'index.html' }),
},
};
@@ -0,0 +1,8 @@
import { sveltekit } from '@sveltejs/kit/vite'
/** @type {import('vite').UserConfig} */
const config = {
plugins: [sveltekit()]
}
export default config
+38 -53
View File
@@ -1,67 +1,52 @@
services:
unifi:
image: lscr.io/linuxserver/unifi-network-application:latest
container_name: unifi
unifi-os-server:
image: ghcr.io/lemker/unifi-os-server:latest
container_name: unifi-os-server
restart: unless-stopped
cgroup: host
cap_add:
- NET_RAW
- NET_ADMIN
extra_hosts:
host.docker.internal: host-gateway
tmpfs:
- /run:exec
- /run/lock
- /tmp:exec
- /var/lib/journal
- /var/opt/unifi/tmp:size=64m
environment:
- PUID=1000
- PGID=1000
- TZ=America/Chicago
- MONGO_USER=${UNIFI_MONGO_USER}
- MONGO_PASS=${UNIFI_MONGO_PASS}
- MONGO_HOST=unifi-db
- MONGO_PORT=27017
- MONGO_DBNAME=unifi
- UOS_SYSTEM_IP=192.168.86.100
volumes:
- /mnt/docker-data/configs/unifi/data:/config
- /sys/fs/cgroup:/sys/fs/cgroup:rw
# /persistent on NAS (config backups, survives reinstall)
- /mnt/docker-data/configs/unifi-os/persistent:/persistent
# All other dirs local — NFS root_squash blocks the chown calls these services need
- /var/lib/unifi-os/var-log:/var/log
- /var/lib/unifi-os/data:/data
- /var/lib/unifi-os/srv:/srv
- /var/lib/unifi-os/var-lib-unifi:/var/lib/unifi
- /var/lib/unifi-os/var-lib-mongodb:/var/lib/mongodb
- /var/lib/unifi-os/etc-rabbitmq-ssl:/etc/rabbitmq/ssl
ports:
- "8443:8443"
- "3478:3478/udp"
- "10001:10001/udp"
- "11443:443"
- "8080:8080"
- "1900:1900/udp"
- "8843:8843"
- "8880:8880"
- "3478:3478/udp"
- "10003:10003/udp"
- "6789:6789"
- "8444:8444"
- "8880:8880"
- "8881:8881"
- "8882:8882"
- "5671:5671"
- "5514:5514/udp"
depends_on:
unifi-db:
condition: service_healthy
networks:
- unifi-internal
mem_limit: 1G
memswap_limit: 1G
logging:
driver: "json-file"
options:
max-size: "50m"
max-file: "3"
unifi-db:
image: mongo:7.0
container_name: unifi-db
restart: unless-stopped
user: "1024:100"
volumes:
- /mnt/docker-data/databases/unifi-db:/data/db
networks:
- unifi-internal
- "9543:9543"
- "5005:5005"
- "127.0.0.1:11084:11084"
labels:
- "com.centurylinklabs.watchtower.enable=false"
mem_limit: 1G
memswap_limit: 1G
healthcheck:
test: ["CMD-SHELL", "mongosh --eval 'db.adminCommand(\"ping\")' --quiet"]
interval: 30s
timeout: 10s
retries: 3
start_period: 30s
logging:
driver: "json-file"
options:
max-size: "50m"
max-size: "100m"
max-file: "3"
networks:
unifi-internal:
driver: bridge
@@ -0,0 +1,303 @@
# LinkedStorm — Home Audio Assistant Design
**Date:** 2026-05-27
**Status:** Approved
---
## Context
The goal is a whole-home audio assistant that passively records household conversations, identifies who is speaking, produces searchable transcripts, and maintains a daily activity log. The motivation is to have a reliable, private, locally-hosted record of decisions and conversations that can be queried naturally ("what did we decide about the vacation?") without relying on cloud services.
The system reuses as much existing homelab infrastructure as possible: Whisper STT on the GPU machine, LiveKit for WebRTC, Ollama for LLM queries, and SpeechBrain for audio ML — all already deployed. Speaker labeling mirrors the proven frigate-notify person-identification pattern (CLIP embeddings → voice embeddings, cosine similarity, three-tier matching, review-and-tag UI).
---
## Architecture
**Hybrid approach:** Lightweight WebSocket audio capture from browser tabs → server-side processing pipeline → LiveKit for live monitoring and replay.
### Container Breakdown
Every deployable component is its own container, following the same pattern as VoiceCoach (7 containers) and the rest of the homelab. LinkedStorm is 3 containers:
| Container | Purpose | Port | Host |
|---|---|---|---|
| `linkedstorm-pipeline` | WebSocket ingestion, Silero VAD, Whisper calls, resemblyzer speaker embedding, cosine similarity matching, SQLite writes, LiveKit publish | 8300 | hf-pdocker-01 |
| `linkedstorm-api` | FastAPI web dashboard + REST API, static UI, FTS5 search, RAG queries, speaker labeling | 8310 | hf-pdocker-01 |
| `linkedstorm-cron` | Nightly Ollama summary, unknown speaker clustering, WAV clip purge | — (no port) | hf-pdocker-01 |
All three share a single Docker volume for the SQLite database and clips directory.
### External Services (Reused Without Modification)
| Service | What | Where |
|---|---|---|
| Speaches/Whisper | `POST /v1/audio/transcriptions` | GPU machine :8000 |
| LiveKit | Per-room replay rooms (`linkedstorm-{room}`) | GPU machine :7880 |
| Ollama | RAG queries + daily summaries (`qwen2.5:14b`) | GPU machine :11434 |
Speaker voice embedding uses `resemblyzer` (ECAPA-TDNN, ~50MB) running in-process inside `linkedstorm-pipeline` — no separate microservice, no coupling to VoiceCoach.
### Data Flow
```
Browser tab (capture.html)
└─ MediaRecorder → WebSocket → linkedstorm-pipeline:8300
└─ Silero VAD detects speech segment
└─ [Whisper STT via Speaches ‖ resemblyzer embed] concurrent
└─ cosine similarity → speaker match
└─ SQLite write + WAV clip save
└─ LiveKit publish → dashboard replay
linkedstorm-api:8310 ← reads SQLite → serves dashboard + API
linkedstorm-cron ← runs nightly at midnight via entrypoint sleep loop
```
---
## Components
### 1. Browser Capture Page (`linkedstorm-capture`)
Single HTML page opened on any device to make it a room microphone.
- `MediaRecorder` at 16kHz mono, 250ms chunks streamed over WebSocket
- Room name set via URL param (`?room=Kitchen`) or prompt on first load
- Persistent room name stored in `localStorage`
- Mute toggle: stops `MediaRecorder`, sends `{"type":"mute"}` control message, server discards buffered audio for that room
- Visual indicator: green pulsing dot (streaming) / red dot (muted) / gray (disconnected)
- Auto-reconnect with exponential backoff on WebSocket drop
### 2. Audio Ingestion Service (`linkedstorm-ingestion`)
Python asyncio service on hf-pdocker-01. Single process, per-room coroutines.
- WebSocket endpoint: `ws://linkedstorm:8300/ws/{room_id}`
- Per-room `asyncio.Queue` of audio chunks
- **Silero VAD** (reused from VoiceCoach): processes each 250ms chunk; accumulates on speech-detected, emits segment on ≥600ms silence; discards segments <0.5s
- On segment emit: calls `process_utterance(room_id, audio_bytes, start_time, end_time)`
- Room registry: upsert `rooms` table on first connection, update `last_seen` on each chunk, set `is_active=False` on disconnect
### 3. Processing Pipeline
Runs as an async task per emitted speech segment.
**Step 1 — Concurrent inference** (asyncio gather):
- `POST http://192.168.86.150:8000/v1/audio/transcriptions` with audio WAV → transcript text
- `resemblyzer.preprocess_wav()` + `encoder.embed_utterance()` in-process → voice embedding (256-dim float32 array)
**Step 2 — Speaker matching:**
- Load all `voice_embeddings` rows from SQLite
- Compute cosine similarity between utterance embedding and each reference
- Group by `speaker_id`, take MAX per speaker
- Thresholds: ≥0.85 → `known`, 0.60–0.84 → `ambiguous`, <0.60 → `unknown`
**Step 3 — Persist:**
- Save audio clip to `/data/linkedstorm/clips/{date}/{room_id}/{utterance_id}.wav`
- Insert `utterances` row: transcript, embedding, room_id, speaker_id, match_status, match_confidence, start_time, end_time, audio_clip_path
**Step 4 — LiveKit publish:**
- Publish audio segment to LiveKit room `linkedstorm-{room_id}` for live monitoring/replay
### 4. Web UI Service (`linkedstorm-ui`)
FastAPI on hf-pdocker-01 port 8310. Static HTML/JS frontend, JSON API backend. Same pattern as frigate-notify.
**API endpoints:**
```
GET /api/utterances # list with filters: date, room_id, speaker_id, match_status
GET /api/utterances/{id}/clip # stream audio clip WAV
POST /api/utterances/{id}/tag # label utterance: {speaker_id} or {name, create_new: true}
POST /api/utterances/{id}/dismiss # mark as noise/unknown, no review needed
GET /api/speakers # list all known speakers with utterance counts
POST /api/speakers # create new speaker
GET /api/rooms # list rooms with is_active, last_seen
POST /api/rooms/{id}/mute # send mute command to room's WebSocket connection
GET /api/search?q=... # full-text search across utterances.transcript (SQLite FTS5)
POST /api/ask # RAG query: builds context from utterances, calls Ollama
GET /api/summary/{date} # fetch or trigger daily summary for a date
```
**UI screens (5 tabs, single-page app):**
1. **Daily Log** — timeline grouped by time blocks, color-coded by speaker, inline Play/Tag/Dismiss on unknowns
2. **Search** — full-text with keyword highlighting, date-range filter
3. **Speakers** — known speaker cards + review-queue banner for unreviewed utterances
4. **Ask AI** — RAG chat backed by Ollama, responses cite room + timestamp sources
5. **Rooms** — live capture device status, per-room mute toggle, link to open capture page
### 5. Nightly Cron Jobs
Run at midnight via cron on hf-pdocker-01.
**Daily summary:** Fetch all utterances for the day → batch into prompt → send to Ollama (`qwen2.5:14b`) → store in `daily_summaries`. Prompt instructs the model to summarize key decisions, activities, and topics — not to transcribe verbatim.
**Unknown speaker clustering:** Load all `unknown` and `ambiguous` utterance embeddings from the past 7 days → cluster by cosine similarity (threshold 0.70) → group consistent voices as `Unknown #1`, `Unknown #2`, etc. in the review queue rather than scattered individuals.
---
## Data Model
SQLite at `/data/linkedstorm/linkedstorm.db`. WAL mode, foreign keys enabled. Thread-local connections (same pattern as frigate-notify).
```sql
CREATE TABLE rooms (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
device_label TEXT,
is_active BOOLEAN DEFAULT 0,
last_seen DATETIME,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE speakers (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
updated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE voice_embeddings (
id INTEGER PRIMARY KEY,
speaker_id INTEGER NOT NULL REFERENCES speakers(id),
embedding BLOB NOT NULL, -- packed float32
audio_sample BLOB, -- short reference WAV
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE utterances (
id INTEGER PRIMARY KEY,
room_id INTEGER NOT NULL REFERENCES rooms(id),
speaker_id INTEGER REFERENCES speakers(id), -- nullable
transcript TEXT NOT NULL,
embedding BLOB, -- packed float32
match_status TEXT DEFAULT 'unreviewed', -- known/unknown/ambiguous/unreviewed
match_confidence REAL,
start_time DATETIME NOT NULL,
end_time DATETIME NOT NULL,
audio_clip_path TEXT,
livekit_room TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE VIRTUAL TABLE utterances_fts USING fts5(
transcript, content=utterances, content_rowid=id
);
CREATE TABLE daily_summaries (
id INTEGER PRIMARY KEY,
date DATE NOT NULL UNIQUE,
summary_text TEXT NOT NULL,
generated_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
```
---
## Speaker Identification
Mirrors frigate-notify's CLIP embedding pattern, adapted for voice:
| Similarity | Status | Action |
|---|---|---|
| ≥ 0.85 | `known` | Auto-label speaker. Store utterance embedding as additional reference. |
| 0.60–0.84 | `ambiguous` | Store utterance. Appear in review queue with Play + Tag buttons. |
| < 0.60 | `unknown` | Store utterance. Cluster with other unknowns nightly. |
| No embed | `unreviewed` | SpeechBrain unavailable. Transcript stored, embed missing. |
**Labeling flow (same as frigate-notify tag workflow):**
`POST /api/utterances/{id}/tag` with `{speaker_id}` or `{name, create_new: true}` →
sets `match_status='known'`, `match_confidence=1.0`, stores utterance's embedding in `voice_embeddings` for future matching.
---
## Deployment
**3 containers on hf-pdocker-01**
**Compose file:** `compose/hf-pdocker-01/linkedstorm/docker-compose.yml`
**Ports:**
- `8300` — `linkedstorm-pipeline` WebSocket ingestion (capture devices connect here)
- `8310` — `linkedstorm-api` FastAPI web UI and API
**Volumes:**
- `/data/linkedstorm/` → `/mnt/docker-data/configs/linkedstorm/` (NAS)
- `linkedstorm.db` — SQLite database
- `clips/` — audio clip WAV files
**Clip retention:** A VAD-triggered household produces roughly 50–200MB of WAV clips per day. The nightly cron should delete clips older than 30 days (configurable via `CLIP_RETENTION_DAYS` env var) while keeping the transcript and embedding rows permanently. The `audio_clip_path` column is set to `NULL` for purged clips; the Play button in the UI shows "clip expired" for those utterances.
**Environment:**
```
SPEACHES_URL=http://192.168.86.150:8000
CLIP_RETENTION_DAYS=30
OLLAMA_URL=http://192.168.86.150:11434
LIVEKIT_URL=ws://192.168.86.150:7880
LIVEKIT_API_KEY=...
LIVEKIT_API_SECRET=...
```
**Traefik:** `linkedstorm.hobbs.farm` → port 8310 (dashboard). Capture page accessed at `linkedstorm.hobbs.farm/capture?room=Kitchen`.
**Cron entries on hf-pdocker-01:**
```
0 0 * * * docker exec linkedstorm python -m cron.daily_summary
5 0 * * * docker exec linkedstorm python -m cron.cluster_unknowns
10 0 * * * docker exec linkedstorm python -m cron.purge_clips
```
---
## File Structure
```
compose/hf-pdocker-01/linkedstorm/
├── docker-compose.yml
├── .env.example
├── pipeline/
│ └── Dockerfile # linkedstorm-pipeline image
├── api/
│ └── Dockerfile # linkedstorm-api image
└── cron/
└── Dockerfile # linkedstorm-cron image
# Shared app code (mounted into pipeline + api containers, or baked into images)
app/
├── config.py # Settings from env vars (shared)
├── database.py # SQLite thread-local connections, schema init (shared)
├── models.py # Dataclasses for rooms, speakers, utterances (shared)
│
├── pipeline/ # linkedstorm-pipeline
│ ├── main.py # asyncio entrypoint: WebSocket server + per-room VAD loops
│ ├── ingestion.py # WebSocket handler, per-room audio buffer + Silero VAD
│ ├── processor.py # process_utterance: Whisper + resemblyzer + matching + persist
│ ├── recognition.py # resemblyzer embed + cosine similarity, speaker matching
│ └── livekit.py # Publish audio segments to LiveKit rooms
│
├── api/ # linkedstorm-api
│ ├── main.py # FastAPI entrypoint, mounts routes, serves static files
│ ├── routes/
│ │ ├── utterances.py # /api/utterances CRUD + tag + dismiss
│ │ ├── speakers.py # /api/speakers CRUD
│ │ ├── rooms.py # /api/rooms status + mute relay to pipeline
│ │ ├── search.py # /api/search (SQLite FTS5)
│ │ └── rag.py # /api/ask, /api/summary
│ └── static/
│ ├── index.html # Dashboard SPA (5-tab layout)
│ ├── capture.html # Minimal capture page (one per room device)
│ └── app.js / style.css
│
└── cron/ # linkedstorm-cron
├── main.py # Sleep-loop entrypoint: runs all jobs at midnight
├── daily_summary.py # Ollama daily summary generation
├── cluster_unknowns.py # Nightly unknown speaker clustering
└── purge_clips.py # Delete WAV files older than CLIP_RETENTION_DAYS
```
---
## Verification
1. **Capture → ingestion:** Open `capture.html?room=Kitchen` in a browser, speak — confirm WebSocket connects, room appears active in the Rooms tab of the dashboard
2. **VAD:** Speak briefly then stay silent — confirm a single utterance appears in the log (not multiple fragments), and background noise is not logged
3. **Transcription:** Read a known sentence — confirm transcript text in the utterance matches
4. **Speaker labeling:** Tag an unknown utterance as "Jeremy" → speak again → confirm next utterances auto-resolve to Jeremy with high confidence
5. **Mute:** Click mute on the capture page — confirm no utterances logged while muted, muted status shown in Rooms tab
6. **Search:** Search for a word spoken earlier — confirm it appears highlighted in results
7. **Ask AI:** Ask "What did we talk about today?" — confirm Ollama returns a coherent summary with room/time citations
8. **LiveKit replay:** From the dashboard, join a room's LiveKit channel — confirm processed audio plays back
9. **Nightly cron:** Trigger `daily_summary` manually — confirm `daily_summaries` row appears and is shown in the log view
10. **Prometheus probe:** Add `http://192.168.86.192:8310/health` to blackbox targets and confirm it's green
+7 -6
View File
@@ -118,27 +118,28 @@ groups:
summary: "{{ $labels.hostname }} rebooted"
description: "{{ $labels.hostname }} was rebooted less than 10 minutes ago."
# High swap usage
# High swap usage - uses absolute free swap threshold to avoid node-exporter v1.10.x
# SwapTotal double-counting bug on swapfile systems. Triggers when free swap < 1.5GB.
- alert: HighSwapUsage
expr: (node_memory_SwapTotal_bytes > 0) and ((node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / node_memory_SwapTotal_bytes * 100 > 50)
expr: (node_memory_SwapTotal_bytes > 0) and (node_memory_SwapFree_bytes < 1.5 * 1024 * 1024 * 1024)
for: 5m
labels:
severity: warning
type: server
annotations:
summary: "{{ $labels.hostname }}: High swap usage"
description: "Swap usage on {{ $labels.hostname }} is above 50% for 5 minutes. Check container memory limits."
description: "Swap usage on {{ $labels.hostname }} is high (free swap below 1.5GB for 5 minutes). Check container memory limits."
# Critical swap usage
# Critical swap usage - triggers when free swap < 512MB
- alert: CriticalSwapUsage
expr: (node_memory_SwapTotal_bytes > 0) and ((node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / node_memory_SwapTotal_bytes * 100 > 80)
expr: (node_memory_SwapTotal_bytes > 0) and (node_memory_SwapFree_bytes < 512 * 1024 * 1024)
for: 2m
labels:
severity: critical
type: server
annotations:
summary: "{{ $labels.hostname }}: Critical swap usage"
description: "Swap usage on {{ $labels.hostname }} is above 80%. System may become unresponsive."
description: "Swap usage on {{ $labels.hostname }} is critical (free swap below 512MB). System may become unresponsive."
# NFS mount failure (all data is on NAS - this is critical)
- alert: NFSMountDown
+15
View File
@@ -0,0 +1,15 @@
http:
routers:
linkedstorm:
rule: "Host(`linkedstorm.hobbs.farm`)"
entryPoints:
- websecure
service: linkedstorm
tls:
certResolver: letsencrypt
services:
linkedstorm:
loadBalancer:
servers:
- url: "http://192.168.86.192:8310"