jeremy/hobbs-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
infra/lifeos-and-pending-sync
hobbs-iac/VOICECOACH_DEPLOY.md
T
jeremy 266ea11447 feat: FRIDAY Voice stack on the GPU machine; VoiceCoach history 
FRIDAY Voice (speaches GPU STT/TTS + hermes agent + push-to-talk console) on
.150 Docker Desktop, fronted by coach.hobbs.farm. Also captures the VoiceCoach
compose/agent trees and deploy notes it replaced, which had never been committed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 11:19:04 -05:00

1974 lines
66 KiB
Markdown
Raw Permalink Blame History

# VoiceCoach — Homelab Deployment Instructions
### Pipecat + SpeechBrain + Claude + LiveKit + OpenClaw
**Target: hobbs.farm Docker Swarm | Author: Claude | For use with Claude Code**
---
## Requirements Iteration Log
> Claude Code: Do NOT skip this section. It documents why decisions were made and will save you from dead ends.
### Iteration 1 — Core Functional Requirements (Raw)
Initial capture of what this system must do:
- Receive real-time voice from Jeremy's phone while driving
- Transcribe speech to text
- Analyze vocal characteristics (pitch, energy, emotion, speaking rate)
- Pass both transcript AND vocal metadata to Claude
- Claude responds with coaching feedback delivered as spoken audio
- Session memory persists across commutes
- Self-hosted on hobbs.farm infrastructure
**Gap identified:** No interface specified. Matrix is async/text-focused — not suitable for real-time bidirectional audio streaming. Need a WebRTC solution.
---
### Iteration 2 — Infrastructure Constraint Pass
Applying Jeremy's known infrastructure to the requirements:
- **Must deploy as Docker Swarm stack** — nodes: hf-pdocker-01, hf-pdocker-02, bart
- **Must use Traefik** for SSL termination with existing wildcard `*.hobbs.farm` cert via Cloudflare DNS challenge
- **Must integrate with OpenClaw** — new agent instance, not a standalone app
- **Must store conversation memory** in OpenClaw's memory system (compatible with Pluto)
- **Interface decision:** Replace Matrix voice with **LiveKit** (self-hosted WebRTC) at `voice.hobbs.farm`, web client at `coach.hobbs.farm`. Matrix retained for text/async agent interaction only.
- **SpeechBrain must run as its own microservice** — not embedded in pipeline — so it can be reused and scaled independently
- **No GPU assumption for initial deploy** — SpeechBrain runs on CPU. GPU path documented but optional.
**Gap identified:** Need to define the exact data contract between SpeechBrain service and Pipecat pipeline. What fields get passed to Claude and in what format.
---
### Iteration 3 — SpeechBrain & Emotion Pipeline Detail
Defining the data contract and model choices:
**SpeechBrain Model:** `speechbrain/emotion-recognition-wav2vec2-IEMOCAP`
- Classifies: neutral, happy, sad, angry, frustrated
- Input: 16kHz mono WAV, minimum ~1 second
- Output: emotion label + per-class confidence scores
**Additional acoustic features via librosa:**
- `pitch_mean_hz` — fundamental frequency average
- `pitch_variance` — pitch stability (high variance = animated/stressed)
- `energy_rms` — loudness/intensity
- `speaking_rate_syllables_per_sec` — estimated via onset detection
- `voiced_fraction` — ratio of voiced to unvoiced frames (low = hesitant/uncertain)
**Prompt injection format (into every Claude turn):**
```
[VOCAL_STATE emotion=frustrated confidence=0.82 pitch=215hz pitch_var=high energy=0.71 rate=4.8sps voiced=0.91]
User said: "I just cannot get the team to listen to me in standups"
```
**Claude system prompt must instruct:**
- Interpret `[VOCAL_STATE]` tags as paralinguistic context
- Distinguish between what the user says and how they sound saying it
- Provide coaching on both content and delivery when relevant
- Never mention the raw numbers — translate to natural coaching language
**Gap identified:** How does Pipecat know when an utterance is complete to trigger SpeechBrain? Need end-of-turn detection strategy. Deepgram handles this via `is_final` events in streaming mode.
---
### Iteration 4 — Testing, Validation, Security
Defining what "working" means before writing code:
**Unit Tests:**
- `test_speechbrain_service.py` — POST audio file, assert emotion field present, confidence > 0, latency < 2s
- `test_pipecat_pipeline.py` — mock Deepgram + SpeechBrain, assert Claude receives correctly formatted prompt
- `test_openclaw_agent.py` — assert agent responds and memory is written after session
**Integration Tests:**
- `test_e2e_pipeline.sh` — send WAV file to LiveKit room via livekit-cli, assert TTS audio received back
- `test_latency.sh` — measure total roundtrip from audio-in to audio-out, assert < 4s
**Docker Health Checks:**
- SpeechBrain service: `GET /health` returns 200, model loaded
- Pipecat agent: process alive check
- LiveKit: `GET /rtc/health`
**Validation Steps (run before marking deploy complete):**
1. `docker service ls` — all 6 services showing replicas as desired
2. `curl https://voice.hobbs.farm/rtc/health` — LiveKit responding
3. `curl https://coach.hobbs.farm` — web client loads
4. SpeechBrain smoke test with provided sample audio
5. Full end-to-end: open coach.hobbs.farm on phone, say "test", hear Claude respond
**Security:**
- LiveKit API key/secret stored as Docker Swarm secrets (not env vars)
- Deepgram API key as Swarm secret
- Anthropic API key as Swarm secret
- ElevenLabs API key as Swarm secret
- coach.hobbs.farm protected by basic auth via Traefik middleware (single user)
- SpeechBrain service NOT exposed externally — internal Swarm overlay network only
- OpenClaw agent token as Swarm secret
**Gap identified:** Need to address baseline calibration — SpeechBrain trained on IEMOCAP actors, not Jeremy's voice. First session should establish Jeremy's neutral baseline so relative changes are meaningful.
---
### Iteration 5 — Memory, Continuity & Calibration
Addressing the coaching quality requirements:
**Baseline Calibration (first session only):**
- First 60 seconds of first session: Claude prompts Jeremy to read a neutral calibration script
- SpeechBrain readings stored in OpenClaw memory as `vocal_baseline` profile
- Subsequent sessions: all readings compared to baseline, delta reported to Claude
- Format: `[VOCAL_STATE emotion=frustrated delta_from_baseline: pitch+18hz energy+0.12]`
**Session Memory Structure in OpenClaw:**
```json
{
"session_id": "2026-04-01-commute-am",
"duration_minutes": 22,
"topics_discussed": ["team communication", "standup facilitation"],
"vocal_trends": {
"avg_stress_level": 0.71,
"peak_frustration_at": "12:34",
"improvement_noted": false
},
"coaching_notes": "Jeremy shows elevated pitch when discussing team dynamics",
"action_items": ["Practice lower speaking rate in next standup"]
}
```
**Cross-session coaching continuity:**
- OpenClaw system prompt includes last 3 session summaries
- Claude tracks whether previous action items were attempted
- Vocal improvement over time is explicitly tracked and acknowledged
**Commute-optimized coaching modes (user-selectable via wake phrase):**
- `"coaching mode"` — deep conversation, feedback on every utterance
- `"vent mode"` — Claude listens, reflects back, minimal interruption
- `"practice mode"` — Claude plays a difficult stakeholder, Jeremy practices
- `"briefing mode"` — Claude summarizes pending tasks from Pluto/OpenClaw memory
**Gap identified:** Need to define how mode switching works mid-conversation without typing. Answer: wake phrases detected at STT layer, routed before Claude prompt.
---
### Iteration 6 — Latency Budget, Fallbacks & Observability
Ensuring the system is actually usable in a moving truck:
**Latency Budget (target < 3.5s total roundtrip):**
- Deepgram streaming STT end-of-turn detection: ~200ms
- SpeechBrain analysis (async, overlaps with STT): ~800ms on CPU
- Claude API call: ~1200ms
- ElevenLabs TTS first audio chunk: ~400ms
- LiveKit transmission: ~100ms
- **Total: ~2.7s** — within target
**Fallbacks:**
- If SpeechBrain service unreachable: pipeline continues without vocal metadata, logs warning
- If Deepgram fails: fallback to Whisper local STT (slower but self-contained)
- If ElevenLabs fails: fallback to edge-tts (local, lower quality)
- If Claude API fails: return "I'm having trouble connecting, please try again"
**Observability:**
- All services log to Docker's logging driver
- Pipecat pipeline emits structured JSON logs per utterance including latencies
- SpeechBrain service logs emotion distribution per session (useful for trend analysis)
- Consider forwarding to existing monitoring stack if Prometheus/Grafana present
**Mobile network resilience:**
- LiveKit handles packet loss via WebRTC NACK/FEC automatically
- Configure Deepgram with `endpointing=500` (500ms silence = end of utterance) — more forgiving for noisy truck cab
- Push-to-talk fallback mode available via coach.hobbs.farm UI (hold button while speaking)
---
### Iteration 7 — Claude Code Readiness Pass
Final pass ensuring Claude Code can execute this without ambiguity:
**Explicit file structure defined** — Claude Code should create exactly these files
**All environment variables listed** with descriptions and example values
**Each service has its own section** with Dockerfile, config, and test
**Deploy order specified** — secrets → network → redis → speechbrain → livekit → pipecat → openclaw → web client
**Rollback procedure documented**
**Every `docker stack deploy` command is copy-pasteable**
**Tests are runnable with a single command after deploy**
---
## System Architecture
```
┌─────────────────────────────────────────────────────────┐
│ PHONE (in truck) │
│ coach.hobbs.farm (Chrome/Safari browser) │
│ LiveKit JS SDK — WebRTC audio stream │
└──────────────────────┬──────────────────────────────────┘
│ WebRTC (DTLS/SRTP)
▼
┌─────────────────────────────────────────────────────────┐
│ LiveKit Server (voice.hobbs.farm) │
│ WebRTC SFU — routes audio frames │
└──────────┬───────────────────────────────────┬──────────┘
│ LiveKit SDK (internal) │ LiveKit SDK
▼ ▼
┌──────────────────────┐ ┌──────────────────────┐
│ Pipecat Pipeline │ │ (future: recording) │
│ Agent Process │ └──────────────────────┘
│ │
│ ┌────────────────┐ │
│ │ Deepgram STT │ │ ← streaming audio chunks
│ │ (streaming) │ │
│ └───────┬────────┘ │
│ │ transcript │
│ ▼ │
│ ┌────────────────┐ │
│ │ SpeechBrain │──┼──→ POST /analyze (internal overlay network)
│ │ Emotion Client │ │ ← {emotion, pitch, energy, rate, voiced}
│ └───────┬────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────┐ │
│ │ Claude API │ │ ← formatted prompt with [VOCAL_STATE]
│ │ claude-sonnet │ │ → coaching response text
│ └───────┬────────┘ │
│ │ │
│ ▼ │
│ ┌────────────────┐ │
│ │ ElevenLabs TTS │ │
│ │ (streaming) │ │
│ └───────┬────────┘ │
│ │ audio │
└──────────┼────────────┘
│ audio frames back to LiveKit
▼
Phone speaker
┌──────────────────────────────────────────────┐
│ SpeechBrain Microservice │
│ (internal network only) │
│ FastAPI + wav2vec2-IEMOCAP + librosa │
│ POST /analyze → emotion + prosody JSON │
│ GET /health → 200 OK │
│ GET /calibrate → session baseline │
└──────────────────────────────────────────────┘
┌──────────────────────────────────────────────┐
│ OpenClaw VoiceCoach Agent │
│ - Manages session lifecycle │
│ - Writes memory after each session │
│ - Exposes /session/start, /session/end │
│ - Integrates with Pluto's memory store │
└──────────────────────────────────────────────┘
```
---
## Repository Structure
Claude Code should create this exact directory structure:
```
voicecoach/
├── docker-stack.yml # Main Swarm stack definition
├── .env.example # Template for secrets (never commit .env)
├── Makefile # Convenience targets
│
├── speechbrain-service/
│ ├── Dockerfile
│ ├── requirements.txt
│ ├── main.py # FastAPI app
│ ├── analyzer.py # SpeechBrain + librosa logic
│ ├── models/ # Model cache (mount as volume)
│ └── tests/
│ ├── test_analyzer.py
│ └── fixtures/
│ ├── neutral_sample.wav
│ ├── angry_sample.wav
│ └── stressed_sample.wav
│
├── pipecat-agent/
│ ├── Dockerfile
│ ├── requirements.txt
│ ├── main.py # Pipecat pipeline entry
│ ├── pipeline.py # Pipeline assembly
│ ├── prompt_builder.py # Builds Claude prompts with vocal state
│ ├── mode_detector.py # Wake phrase detection
│ ├── openclaw_client.py # Writes session memory to OpenClaw
│ └── tests/
│ ├── test_pipeline.py
│ └── test_prompt_builder.py
│
├── openclaw-agent/
│ ├── openclaw.json # OpenClaw agent config
│ ├── system_prompt.md # VoiceCoach persona & instructions
│ └── skills/
│ └── vocal_memory.md # Skill for reading/writing vocal baselines
│
├── web-client/
│ ├── Dockerfile # nginx serving static files
│ ├── index.html
│ ├── app.js # LiveKit JS SDK integration
│ └── style.css
│
├── livekit/
│ └── livekit.yaml # LiveKit server config
│
├── scripts/
│ ├── create-secrets.sh # Creates all Docker Swarm secrets
│ ├── smoke-test.sh # Quick validation after deploy
│ ├── e2e-test.sh # Full end-to-end test
│ └── calibrate.sh # Trigger baseline calibration session
│
└── tests/
└── test_latency.sh # Roundtrip latency measurement
```
---
## Step 0: Prerequisites Checklist
Before Claude Code runs any deployment commands, verify:
```bash
# 1. Confirm Swarm manager access
docker node ls
# 2. Confirm Traefik is running with wildcard cert
docker service ls | grep traefik
# 3. Confirm overlay network exists (or create it)
docker network ls | grep hobbs-overlay
# If missing:
docker network create --driver overlay --attachable hobbs-overlay
# 4. Confirm required external API keys are available
# - Anthropic API key (claude-sonnet-4-6)
# - Deepgram API key (https://console.deepgram.com)
# - ElevenLabs API key (https://elevenlabs.io)
# - LiveKit will use self-generated key pair (no external account needed)
# 5. Confirm ports available
# - 7880/tcp LiveKit HTTPS API
# - 7881/tcp LiveKit RTC over TCP fallback
# - 50000-60000/udp LiveKit WebRTC media (must be open on firewall/router)
```
**IMPORTANT:** If ports 50000-60000 UDP are not open at the router level, WebRTC media will not flow. This is the most common failure point. Confirm with:
```bash
# From external network (not homelab):
nmap -sU -p 50000-50010 <your-public-ip>
```
---
## Step 1: Create Docker Swarm Secrets
**File: `scripts/create-secrets.sh`**
```bash
#!/bin/bash
# Run this ONCE on the Swarm manager node.
# Have your API keys ready before running.
set -e
echo "Creating Docker Swarm secrets for VoiceCoach..."
read -s -p "Anthropic API Key: " ANTHROPIC_KEY && echo
read -s -p "Deepgram API Key: " DEEPGRAM_KEY && echo
read -s -p "ElevenLabs API Key: " ELEVENLABS_KEY && echo
read -s -p "ElevenLabs Voice ID: " ELEVENLABS_VOICE_ID && echo
read -s -p "OpenClaw Agent Token: " OPENCLAW_TOKEN && echo
# Generate LiveKit key pair
LIVEKIT_API_KEY="voicecoach-$(openssl rand -hex 4)"
LIVEKIT_API_SECRET=$(openssl rand -base64 32)
echo "$ANTHROPIC_KEY" | docker secret create voicecoach_anthropic_key -
echo "$DEEPGRAM_KEY" | docker secret create voicecoach_deepgram_key -
echo "$ELEVENLABS_KEY" | docker secret create voicecoach_elevenlabs_key -
echo "$ELEVENLABS_VOICE_ID" | docker secret create voicecoach_elevenlabs_voice_id -
echo "$OPENCLAW_TOKEN" | docker secret create voicecoach_openclaw_token -
echo "$LIVEKIT_API_KEY" | docker secret create voicecoach_livekit_api_key -
echo "$LIVEKIT_API_SECRET" | docker secret create voicecoach_livekit_api_secret -
echo ""
echo "✅ Secrets created. LiveKit credentials:"
echo " API Key: $LIVEKIT_API_KEY"
echo " API Secret: $LIVEKIT_API_SECRET"
echo ""
echo "⚠️ Save these LiveKit credentials — they are stored in Swarm secrets"
echo " and cannot be retrieved again."
```
---
## Step 2: SpeechBrain Microservice
### `speechbrain-service/requirements.txt`
```
fastapi==0.115.0
uvicorn[standard]==0.30.0
speechbrain==1.0.0
torch==2.3.0
torchaudio==2.3.0
librosa==0.10.2
soundfile==0.12.1
numpy==1.26.4
httpx==0.27.0
python-multipart==0.0.9
```
### `speechbrain-service/analyzer.py`
```python
"""
SpeechBrain + librosa vocal analysis.
Performs emotion classification and acoustic feature extraction from audio.
"""
import io
import logging
import numpy as np
import soundfile as sf
import librosa
from speechbrain.inference.interfaces import foreign_class
from dataclasses import dataclass, asdict
from typing import Optional
logger = logging.getLogger(__name__)
SAMPLE_RATE = 16000
MIN_DURATION_SECONDS = 0.5 # Reject clips shorter than this
@dataclass
class VocalAnalysis:
emotion: str
emotion_confidence: float
emotion_scores: dict # All class probabilities
pitch_mean_hz: Optional[float]
pitch_variance: Optional[float]
energy_rms: float
speaking_rate_approx: float # Onset events per second (proxy for syllable rate)
voiced_fraction: float # 0-1, fraction of voiced frames
duration_seconds: float
analysis_error: Optional[str] = None
class VocalAnalyzer:
"""Loads models once at startup and provides analysis per audio chunk."""
def __init__(self, model_dir: str = "/app/models"):
logger.info("Loading SpeechBrain emotion model...")
self.classifier = foreign_class(
source="speechbrain/emotion-recognition-wav2vec2-IEMOCAP",
pymodule_file="custom_interface.py",
classname="CustomEncoderWav2vec2Classifier",
savedir=model_dir,
run_opts={"device": "cpu"}, # Change to "cuda" if GPU available
)
logger.info("SpeechBrain model loaded successfully.")
def analyze(self, audio_bytes: bytes, content_type: str = "audio/wav") -> VocalAnalysis:
"""
Analyze audio bytes and return vocal features.
Args:
audio_bytes: Raw audio file bytes (WAV, FLAC, or OGG)
content_type: MIME type hint
Returns:
VocalAnalysis dataclass
"""
try:
# Load audio
audio_buf = io.BytesIO(audio_bytes)
audio, sr = sf.read(audio_buf, dtype="float32")
# Ensure mono
if audio.ndim > 1:
audio = audio.mean(axis=1)
# Resample to 16kHz if needed
if sr != SAMPLE_RATE:
audio = librosa.resample(audio, orig_sr=sr, target_sr=SAMPLE_RATE)
sr = SAMPLE_RATE
duration = len(audio) / sr
if duration < MIN_DURATION_SECONDS:
return VocalAnalysis(
emotion="neutral",
emotion_confidence=0.0,
emotion_scores={},
pitch_mean_hz=None,
pitch_variance=None,
energy_rms=0.0,
speaking_rate_approx=0.0,
voiced_fraction=0.0,
duration_seconds=duration,
analysis_error=f"Audio too short: {duration:.2f}s < {MIN_DURATION_SECONDS}s minimum",
)
# --- SpeechBrain emotion classification ---
import tempfile, os
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
sf.write(tmp.name, audio, sr)
tmp_path = tmp.name
try:
out_prob, score, index, text_lab = self.classifier.classify_file(tmp_path)
emotion = text_lab[0] if isinstance(text_lab, list) else text_lab
confidence = float(score[0]) if hasattr(score, "__iter__") else float(score)
# Build score dict from probability tensor
labels = ["neu", "hap", "sad", "ang", "fru"] # IEMOCAP classes
prob_list = out_prob[0].tolist() if hasattr(out_prob[0], "tolist") else list(out_prob[0])
emotion_scores = {l: round(p, 4) for l, p in zip(labels, prob_list)}
finally:
os.unlink(tmp_path)
# --- Librosa acoustic features ---
# Pitch via pyin (more accurate than yin for speech)
f0, voiced_flag, _ = librosa.pyin(
audio,
fmin=librosa.note_to_hz("C2"),
fmax=librosa.note_to_hz("C7"),
sr=sr,
)
voiced_f0 = f0[voiced_flag] if voiced_flag is not None else np.array([])
pitch_mean = float(np.mean(voiced_f0)) if len(voiced_f0) > 0 else None
pitch_var = float(np.var(voiced_f0)) if len(voiced_f0) > 1 else None
voiced_fraction = float(np.sum(voiced_flag) / len(voiced_flag)) if voiced_flag is not None else 0.0
# Energy
rms = librosa.feature.rms(y=audio)
energy_rms = float(np.mean(rms))
# Speaking rate proxy: onset strength events per second
onset_frames = librosa.onset.onset_detect(y=audio, sr=sr)
speaking_rate = len(onset_frames) / duration if duration > 0 else 0.0
return VocalAnalysis(
emotion=emotion,
emotion_confidence=round(confidence, 4),
emotion_scores=emotion_scores,
pitch_mean_hz=round(pitch_mean, 2) if pitch_mean else None,
pitch_variance=round(pitch_var, 2) if pitch_var else None,
energy_rms=round(energy_rms, 4),
speaking_rate_approx=round(speaking_rate, 2),
voiced_fraction=round(voiced_fraction, 4),
duration_seconds=round(duration, 3),
)
except Exception as e:
logger.exception(f"Analysis failed: {e}")
return VocalAnalysis(
emotion="neutral",
emotion_confidence=0.0,
emotion_scores={},
pitch_mean_hz=None,
pitch_variance=None,
energy_rms=0.0,
speaking_rate_approx=0.0,
voiced_fraction=0.0,
duration_seconds=0.0,
analysis_error=str(e),
)
```
### `speechbrain-service/main.py`
```python
"""
FastAPI service wrapping VocalAnalyzer.
Internal-only — not exposed via Traefik.
"""
import logging
import os
from contextlib import asynccontextmanager
from fastapi import FastAPI, UploadFile, File, HTTPException
from fastapi.responses import JSONResponse
from analyzer import VocalAnalyzer, asdict
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
MODEL_DIR = os.getenv("MODEL_DIR", "/app/models")
analyzer: VocalAnalyzer = None
@asynccontextmanager
async def lifespan(app: FastAPI):
global analyzer
logger.info("Starting VocalAnalyzer...")
analyzer = VocalAnalyzer(model_dir=MODEL_DIR)
logger.info("VocalAnalyzer ready.")
yield
logger.info("Shutting down.")
app = FastAPI(title="SpeechBrain Vocal Analysis Service", lifespan=lifespan)
@app.get("/health")
async def health():
if analyzer is None:
raise HTTPException(status_code=503, detail="Model not loaded")
return {"status": "ok", "model": "emotion-recognition-wav2vec2-IEMOCAP"}
@app.post("/analyze")
async def analyze(audio: UploadFile = File(...)):
"""
Analyze vocal characteristics of an audio clip.
Accepts: WAV, FLAC, OGG audio files.
Returns: Emotion classification + acoustic features.
"""
if analyzer is None:
raise HTTPException(status_code=503, detail="Model not loaded")
content = await audio.read()
if len(content) == 0:
raise HTTPException(status_code=400, detail="Empty audio file")
result = analyzer.analyze(content, content_type=audio.content_type or "audio/wav")
return JSONResponse(content=asdict(result))
@app.post("/calibrate")
async def calibrate(audio: UploadFile = File(...)):
"""
Analyze audio and return it labeled as baseline calibration.
Same as /analyze but semantically marks this as a neutral reference.
"""
result_response = await analyze(audio)
data = result_response.body
import json
result_dict = json.loads(data)
result_dict["is_baseline"] = True
return JSONResponse(content=result_dict)
```
### `speechbrain-service/Dockerfile`
```dockerfile
FROM python:3.11-slim
# System deps for librosa/soundfile
RUN apt-get update && apt-get install -y \
libsndfile1 \
ffmpeg \
git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Pre-download the SpeechBrain model at build time to avoid cold start
# This bakes the model into the image (~400MB)
RUN python -c "
from speechbrain.inference.interfaces import foreign_class
foreign_class(
source='speechbrain/emotion-recognition-wav2vec2-IEMOCAP',
pymodule_file='custom_interface.py',
classname='CustomEncoderWav2vec2Classifier',
savedir='/app/models',
run_opts={'device': 'cpu'},
)
print('Model downloaded successfully.')
"
COPY . .
EXPOSE 8001
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s \
CMD python -c "import httpx; httpx.get('http://localhost:8001/health').raise_for_status()"
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001", "--workers", "1"]
```
### `speechbrain-service/tests/test_analyzer.py`
```python
"""
Tests for the SpeechBrain vocal analyzer.
Requires test WAV fixtures to be present.
"""
import os
import pytest
import numpy as np
import soundfile as sf
from pathlib import Path
FIXTURES = Path(__file__).parent / "fixtures"
def create_sine_wav(path: Path, frequency: float = 440.0, duration: float = 2.0, sr: int = 16000):
"""Generate a sine wave WAV file for testing."""
t = np.linspace(0, duration, int(sr * duration))
audio = 0.5 * np.sin(2 * np.pi * frequency * t).astype(np.float32)
sf.write(str(path), audio, sr)
@pytest.fixture(scope="module", autouse=True)
def create_fixtures():
FIXTURES.mkdir(exist_ok=True)
create_sine_wav(FIXTURES / "neutral_sample.wav", frequency=200, duration=2.0)
create_sine_wav(FIXTURES / "short_sample.wav", frequency=200, duration=0.2)
def test_analyzer_loads():
"""Analyzer must load without error."""
from analyzer import VocalAnalyzer
a = VocalAnalyzer()
assert a is not None
def test_analyze_returns_emotion():
from analyzer import VocalAnalyzer
a = VocalAnalyzer()
audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes()
result = a.analyze(audio_bytes)
assert result.emotion in ["neu", "hap", "sad", "ang", "fru"]
assert 0.0 <= result.emotion_confidence <= 1.0
def test_analyze_returns_acoustic_features():
from analyzer import VocalAnalyzer
a = VocalAnalyzer()
audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes()
result = a.analyze(audio_bytes)
assert result.energy_rms >= 0.0
assert result.duration_seconds > 0.0
assert 0.0 <= result.voiced_fraction <= 1.0
def test_short_audio_handled_gracefully():
from analyzer import VocalAnalyzer
a = VocalAnalyzer()
audio_bytes = (FIXTURES / "short_sample.wav").read_bytes()
result = a.analyze(audio_bytes)
assert result.analysis_error is not None
assert "too short" in result.analysis_error.lower()
def test_analyze_latency():
"""Analysis should complete in under 3 seconds on CPU."""
import time
from analyzer import VocalAnalyzer
a = VocalAnalyzer()
audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes()
start = time.time()
a.analyze(audio_bytes)
elapsed = time.time() - start
assert elapsed < 3.0, f"Analysis took {elapsed:.2f}s — exceeds 3s budget"
```
---
## Step 3: Pipecat Voice Pipeline Agent
### `pipecat-agent/requirements.txt`
```
pipecat-ai[deepgram,elevenlabs,livekit,silero]==0.0.48
anthropic==0.34.0
httpx==0.27.0
python-dotenv==1.0.1
structlog==24.4.0
```
### `pipecat-agent/prompt_builder.py`
```python
"""
Builds Claude prompts with embedded vocal state metadata.
"""
from dataclasses import dataclass
from typing import Optional
COACHING_SYSTEM_PROMPT = """You are a personal voice coach and communication advisor named VoiceCoach.
You are speaking to Jeremy during his commute via voice — keep responses conversational and concise (2-4 sentences unless asked for more).
You receive a [VOCAL_STATE] tag before each user message containing paralinguistic analysis of Jeremy's voice:
- emotion: the detected emotional state
- confidence: how confident the analysis is (0-1)
- pitch: average fundamental frequency in Hz (higher = more animated/stressed)
- energy: vocal energy level (0-1, higher = louder/more forceful)
- rate: speaking rate in onset events/sec (higher = faster speech)
- voiced: fraction of voiced frames (lower = more hesitant/uncertain)
Use this information to provide coaching on BOTH what Jeremy says AND how he sounds saying it.
Important: Never quote the raw numbers. Translate them into natural observations.
Example: "I notice some tension in your voice..." not "Your pitch was 215hz..."
{baseline_context}
{session_history}
Available coaching modes (Jeremy can switch by saying the phrase):
- "coaching mode" (default) — active feedback on content and delivery
- "vent mode" — listen and reflect, minimal advice
- "practice mode" — roleplay a difficult stakeholder
- "briefing mode" — summarize Jeremy's tasks and context from memory
Current mode: {current_mode}
"""
@dataclass
class VocalState:
emotion: str
confidence: float
pitch_mean_hz: Optional[float]
energy_rms: float
speaking_rate_approx: float
voiced_fraction: float
def format_vocal_state(state: VocalState) -> str:
"""Format vocal state as the [VOCAL_STATE] tag injected before each user message."""
pitch_str = f"{state.pitch_mean_hz:.0f}hz" if state.pitch_mean_hz else "unknown"
return (
f"[VOCAL_STATE "
f"emotion={state.emotion} "
f"confidence={state.confidence:.2f} "
f"pitch={pitch_str} "
f"energy={state.energy_rms:.2f} "
f"rate={state.speaking_rate_approx:.1f}ops "
f"voiced={state.voiced_fraction:.2f}]"
)
def build_user_message(transcript: str, vocal_state: Optional[VocalState]) -> str:
"""Combines transcript with vocal state tag."""
if vocal_state and vocal_state.confidence > 0.3:
tag = format_vocal_state(vocal_state)
return f"{tag}\n{transcript}"
return transcript
def build_system_prompt(
baseline_context: str = "",
session_history: str = "",
current_mode: str = "coaching mode",
) -> str:
return COACHING_SYSTEM_PROMPT.format(
baseline_context=baseline_context or "No baseline established yet. First session: ask Jeremy to read a short neutral calibration script.",
session_history=session_history or "No previous session history.",
current_mode=current_mode,
)
```
### `pipecat-agent/pipeline.py`
```python
"""
Main Pipecat pipeline assembly.
Connects LiveKit → Deepgram → SpeechBrain → Claude → ElevenLabs → LiveKit.
"""
import asyncio
import os
import logging
import httpx
from typing import Optional
from pipecat.frames.frames import (
AudioRawFrame, TranscriptionFrame, TextFrame, EndFrame
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.anthropic import AnthropicLLMService
from pipecat.services.deepgram import DeepgramSTTService, LiveOptions
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.transports.services.livekit import LiveKitTransport, LiveKitParams
from pipecat.processors.frame_processor import FrameProcessor, FrameDirection
from pipecat.vad.silero import SileroVADAnalyzer
from prompt_builder import build_system_prompt, build_user_message, VocalState
logger = logging.getLogger(__name__)
SPEECHBRAIN_URL = os.getenv("SPEECHBRAIN_SERVICE_URL", "http://speechbrain-service:8001")
ANTHROPIC_MODEL = "claude-sonnet-4-6"
class VocalStateInjector(FrameProcessor):
"""
Intercepts TranscriptionFrames, fetches vocal analysis from SpeechBrain,
and enriches the frame before passing to the LLM context.
Audio buffer is maintained separately from the STT pipeline —
SpeechBrain receives a copy of the raw audio for each utterance.
"""
def __init__(self, **kwargs):
super().__init__(**kwargs)
self._current_audio: list[bytes] = []
self._http_client = httpx.AsyncClient(timeout=3.0)
async def process_frame(self, frame, direction: FrameDirection):
await super().process_frame(frame, direction)
if isinstance(frame, AudioRawFrame):
# Buffer audio for the current utterance
self._current_audio.append(frame.audio)
elif isinstance(frame, TranscriptionFrame):
# Utterance complete — analyze buffered audio
vocal_state = await self._analyze_buffered_audio()
self._current_audio = [] # Reset buffer
# Inject vocal state into the transcript
enriched_text = build_user_message(frame.text, vocal_state)
enriched_frame = TranscriptionFrame(
text=enriched_text,
user_id=frame.user_id,
timestamp=frame.timestamp,
)
await self.push_frame(enriched_frame, direction)
else:
await self.push_frame(frame, direction)
async def _analyze_buffered_audio(self) -> Optional[VocalState]:
if not self._current_audio:
return None
try:
import io, wave
audio_bytes = b"".join(self._current_audio)
# Wrap raw PCM in WAV container for SpeechBrain
buf = io.BytesIO()
with wave.open(buf, "wb") as wf:
wf.setnchannels(1)
wf.setsampwidth(2) # 16-bit
wf.setframerate(16000)
wf.writeframes(audio_bytes)
wav_bytes = buf.getvalue()
response = await self._http_client.post(
f"{SPEECHBRAIN_URL}/analyze",
files={"audio": ("utterance.wav", wav_bytes, "audio/wav")},
)
response.raise_for_status()
data = response.json()
return VocalState(
emotion=data.get("emotion", "neutral"),
confidence=data.get("emotion_confidence", 0.0),
pitch_mean_hz=data.get("pitch_mean_hz"),
energy_rms=data.get("energy_rms", 0.0),
speaking_rate_approx=data.get("speaking_rate_approx", 0.0),
voiced_fraction=data.get("voiced_fraction", 0.0),
)
except Exception as e:
logger.warning(f"SpeechBrain analysis failed (continuing without): {e}")
return None
async def run_voice_coach_pipeline(
livekit_url: str,
livekit_token: str,
anthropic_key: str,
deepgram_key: str,
elevenlabs_key: str,
elevenlabs_voice_id: str,
session_context: dict,
):
"""
Assemble and run the full voice coaching pipeline.
session_context contains: baseline_context, session_history, current_mode
"""
# Transport: LiveKit WebRTC
transport = LiveKitTransport(
url=livekit_url,
token=livekit_token,
room_name="voicecoach",
params=LiveKitParams(
audio_in_enabled=True,
audio_out_enabled=True,
vad_analyzer=SileroVADAnalyzer(),
),
)
# STT: Deepgram streaming with aggressive endpointing for noisy car
stt = DeepgramSTTService(
api_key=deepgram_key,
live_options=LiveOptions(
model="nova-2",
language="en-US",
endpointing=500, # 500ms silence = end of utterance
interim_results=False,
smart_format=True,
punctuate=True,
),
)
# Vocal state injection (SpeechBrain)
vocal_injector = VocalStateInjector()
# LLM: Claude
llm = AnthropicLLMService(
api_key=anthropic_key,
model=ANTHROPIC_MODEL,
max_tokens=512,
)
# System prompt with context
system_prompt = build_system_prompt(
baseline_context=session_context.get("baseline_context", ""),
session_history=session_context.get("session_history", ""),
current_mode=session_context.get("current_mode", "coaching mode"),
)
context = OpenAILLMContext(
messages=[{"role": "system", "content": system_prompt}]
)
context_aggregator = llm.create_context_aggregator(context)
# TTS: ElevenLabs streaming
tts = ElevenLabsTTSService(
api_key=elevenlabs_key,
voice_id=elevenlabs_voice_id,
model="eleven_turbo_v2_5", # Lowest latency model
optimize_streaming_latency=4,
)
# Assemble pipeline
pipeline = Pipeline([
transport.input(),
stt,
vocal_injector,
context_aggregator.user(),
llm,
tts,
transport.output(),
context_aggregator.assistant(),
])
task = PipelineTask(
pipeline,
params=PipelineParams(allow_interruptions=True),
)
runner = PipelineRunner()
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, participant):
logger.info(f"Participant joined: {participant['identity']}")
await task.queue_frames([
TextFrame("VoiceCoach connected. How are you doing today?")
])
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, participant, reason):
logger.info(f"Participant left: {participant['identity']}")
await task.queue_frames([EndFrame()])
await runner.run(task)
```
### `pipecat-agent/main.py`
```python
"""
Entry point for the Pipecat VoiceCoach agent.
Reads secrets from Docker Swarm secret files.
"""
import asyncio
import logging
import os
from pathlib import Path
import structlog
from livekit import api as livekit_api
from pipeline import run_voice_coach_pipeline
from openclaw_client import OpenClawClient
logging.basicConfig(level=logging.INFO)
logger = structlog.get_logger()
def read_secret(name: str, env_fallback: str = None) -> str:
"""Read from Docker Swarm secret file, fall back to env var."""
secret_path = Path(f"/run/secrets/{name}")
if secret_path.exists():
return secret_path.read_text().strip()
if env_fallback and os.getenv(env_fallback):
return os.getenv(env_fallback)
raise RuntimeError(f"Secret '{name}' not found at {secret_path} and no env fallback")
async def main():
# Load secrets
anthropic_key = read_secret("voicecoach_anthropic_key", "ANTHROPIC_API_KEY")
deepgram_key = read_secret("voicecoach_deepgram_key", "DEEPGRAM_API_KEY")
elevenlabs_key = read_secret("voicecoach_elevenlabs_key", "ELEVENLABS_API_KEY")
elevenlabs_voice_id = read_secret("voicecoach_elevenlabs_voice_id", "ELEVENLABS_VOICE_ID")
livekit_api_key = read_secret("voicecoach_livekit_api_key", "LIVEKIT_API_KEY")
livekit_api_secret = read_secret("voicecoach_livekit_api_secret", "LIVEKIT_API_SECRET")
openclaw_token = read_secret("voicecoach_openclaw_token", "OPENCLAW_TOKEN")
livekit_url = os.getenv("LIVEKIT_URL", "ws://livekit-server:7880")
openclaw_url = os.getenv("OPENCLAW_URL", "http://openclaw-voicecoach:3000")
# Fetch session context from OpenClaw
openclaw = OpenClawClient(openclaw_url, openclaw_token)
session_context = await openclaw.get_session_context()
logger.info("Session context loaded", mode=session_context.get("current_mode"))
# Generate LiveKit access token for this agent
token = livekit_api.AccessToken(livekit_api_key, livekit_api_secret)
token.with_grants(livekit_api.VideoGrants(room_join=True, room="voicecoach"))
token.identity = "voicecoach-agent"
livekit_token = token.to_jwt()
logger.info("Starting VoiceCoach pipeline", livekit_url=livekit_url)
await run_voice_coach_pipeline(
livekit_url=livekit_url,
livekit_token=livekit_token,
anthropic_key=anthropic_key,
deepgram_key=deepgram_key,
elevenlabs_key=elevenlabs_key,
elevenlabs_voice_id=elevenlabs_voice_id,
session_context=session_context,
)
if __name__ == "__main__":
asyncio.run(main())
```
### `pipecat-agent/openclaw_client.py`
```python
"""
Client for reading/writing session data to the OpenClaw VoiceCoach agent.
"""
import httpx
import logging
logger = logging.getLogger(__name__)
class OpenClawClient:
def __init__(self, base_url: str, token: str):
self.base_url = base_url.rstrip("/")
self.headers = {"Authorization": f"Bearer {token}"}
async def get_session_context(self) -> dict:
"""Fetch baseline and recent session history from OpenClaw memory."""
try:
async with httpx.AsyncClient(timeout=5.0) as client:
r = await client.get(
f"{self.base_url}/api/memory",
headers=self.headers,
params={"keys": "vocal_baseline,recent_sessions,current_mode"},
)
r.raise_for_status()
memory = r.json()
return {
"baseline_context": memory.get("vocal_baseline", ""),
"session_history": memory.get("recent_sessions", ""),
"current_mode": memory.get("current_mode", "coaching mode"),
}
except Exception as e:
logger.warning(f"Could not fetch OpenClaw context: {e}. Using defaults.")
return {"baseline_context": "", "session_history": "", "current_mode": "coaching mode"}
async def write_session_summary(self, summary: dict) -> bool:
"""Write end-of-session summary to OpenClaw memory."""
try:
async with httpx.AsyncClient(timeout=5.0) as client:
r = await client.post(
f"{self.base_url}/api/memory",
headers=self.headers,
json={"key": "recent_sessions", "value": summary},
)
r.raise_for_status()
return True
except Exception as e:
logger.error(f"Failed to write session summary: {e}")
return False
```
### `pipecat-agent/Dockerfile`
```dockerfile
FROM python:3.11-slim
RUN apt-get update && apt-get install -y \
libsndfile1 \
ffmpeg \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
HEALTHCHECK --interval=30s --timeout=10s --start-period=30s \
CMD python -c "import asyncio; print('pipeline process alive')" || exit 1
CMD ["python", "main.py"]
```
### `pipecat-agent/tests/test_prompt_builder.py`
```python
"""Tests for prompt assembly."""
from prompt_builder import build_user_message, VocalState, build_system_prompt
def test_vocal_state_injected_into_message():
state = VocalState(
emotion="ang",
confidence=0.82,
pitch_mean_hz=215.0,
energy_rms=0.71,
speaking_rate_approx=4.8,
voiced_fraction=0.91,
)
result = build_user_message("I can't get my team to listen", state)
assert "[VOCAL_STATE" in result
assert "emotion=ang" in result
assert "I can't get my team to listen" in result
def test_low_confidence_vocal_state_omitted():
state = VocalState(
emotion="ang",
confidence=0.15, # Below threshold
pitch_mean_hz=None,
energy_rms=0.0,
speaking_rate_approx=0.0,
voiced_fraction=0.0,
)
result = build_user_message("Hello", state)
assert "[VOCAL_STATE" not in result
def test_none_vocal_state_handled():
result = build_user_message("Hello", None)
assert result == "Hello"
def test_system_prompt_contains_mode():
prompt = build_system_prompt(current_mode="vent mode")
assert "vent mode" in prompt
def test_system_prompt_contains_baseline_when_provided():
prompt = build_system_prompt(baseline_context="Jeremy's baseline pitch: 180hz")
assert "180hz" in prompt
```
---
## Step 4: LiveKit Server Config
### `livekit/livekit.yaml`
```yaml
port: 7880
rtc:
tcp_port: 7881
port_range_start: 50000
port_range_end: 60000
use_external_ip: true # Required for remote clients (phone from road)
# Logging
logging:
json: false
level: info
# Room settings
room:
max_participants: 5
empty_timeout: 300 # Close room after 5min of no participants
departure_timeout: 30
# Keys are loaded from environment — injected via Docker secrets
# Format: key_name: secret_value (loaded from LIVEKIT_KEYS env var)
```
---
## Step 5: Web Client
### `web-client/index.html`
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<title>VoiceCoach</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<div class="container">
<div class="header">
<h1>🎙 VoiceCoach</h1>
<div id="status" class="status disconnected">Disconnected</div>
</div>
<div class="controls">
<button id="connectBtn" class="btn btn-primary">Start Session</button>
<button id="disconnectBtn" class="btn btn-danger" style="display:none">End Session</button>
</div>
<div id="modeDisplay" class="mode-display">Mode: coaching</div>
<div id="transcript" class="transcript">
<p class="hint">Press "Start Session" to connect your voice.</p>
</div>
<div class="mode-buttons">
<button class="mode-btn" onclick="setMode('coaching mode')">🎯 Coaching</button>
<button class="mode-btn" onclick="setMode('vent mode')">💬 Vent</button>
<button class="mode-btn" onclick="setMode('practice mode')">🎭 Practice</button>
<button class="mode-btn" onclick="setMode('briefing mode')">📋 Briefing</button>
</div>
</div>
<script src="https://cdn.jsdelivr.net/npm/livekit-client/dist/livekit-client.umd.min.js"></script>
<script src="app.js"></script>
</body>
</html>
```
### `web-client/app.js`
```javascript
const GATEWAY_URL = window.location.origin + "/api";
let room = null;
async function getToken() {
const response = await fetch(`${GATEWAY_URL}/token`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ identity: "jeremy-" + Date.now() }),
});
if (!response.ok) throw new Error("Failed to get token");
return response.json();
}
async function connect() {
setStatus("connecting", "Connecting...");
try {
const { token, url } = await getToken();
room = new LivekitClient.Room({
audioCaptureDefaults: {
echoCancellation: true,
noiseSuppression: true,
autoGainControl: true,
},
});
room.on(LivekitClient.RoomEvent.Connected, () => {
setStatus("connected", "Connected");
document.getElementById("connectBtn").style.display = "none";
document.getElementById("disconnectBtn").style.display = "block";
});
room.on(LivekitClient.RoomEvent.Disconnected, () => {
setStatus("disconnected", "Disconnected");
document.getElementById("connectBtn").style.display = "block";
document.getElementById("disconnectBtn").style.display = "none";
});
room.on(LivekitClient.RoomEvent.TrackSubscribed, (track) => {
if (track.kind === LivekitClient.Track.Kind.Audio) {
const element = track.attach();
document.body.appendChild(element);
element.play();
}
});
await room.connect(url, token);
await room.localParticipant.setMicrophoneEnabled(true);
} catch (err) {
setStatus("disconnected", "Error: " + err.message);
console.error(err);
}
}
async function disconnect() {
if (room) {
await room.disconnect();
room = null;
}
}
function setMode(mode) {
document.getElementById("modeDisplay").textContent = "Mode: " + mode;
if (room && room.localParticipant) {
// Send mode as data message to agent
const encoder = new TextEncoder();
room.localParticipant.publishData(
encoder.encode(JSON.stringify({ type: "mode_change", mode })),
{ reliable: true }
);
}
}
function setStatus(state, text) {
const el = document.getElementById("status");
el.className = "status " + state;
el.textContent = text;
}
document.getElementById("connectBtn").addEventListener("click", connect);
document.getElementById("disconnectBtn").addEventListener("click", disconnect);
```
### `web-client/Dockerfile`
```dockerfile
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/
COPY app.js /usr/share/nginx/html/
COPY style.css /usr/share/nginx/html/
# Token endpoint proxy config
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
```
### `web-client/nginx.conf`
```nginx
server {
listen 80;
root /usr/share/nginx/html;
index index.html;
# Proxy /api/* to the pipecat agent's token service
location /api/ {
proxy_pass http://pipecat-agent:8080/;
proxy_http_version 1.1;
proxy_set_header Host $host;
}
location / {
try_files $uri $uri/ /index.html;
}
}
```
---
## Step 6: OpenClaw Agent Config
### `openclaw-agent/openclaw.json`
```json
{
"name": "VoiceCoach",
"description": "Real-time voice coaching agent for commute sessions. Integrates vocal analysis with communication coaching.",
"version": "1.0.0",
"model": {
"provider": "anthropic",
"name": "claude-sonnet-4-6",
"temperature": 0.7,
"max_tokens": 512
},
"memory": {
"enabled": true,
"keys": [
"vocal_baseline",
"recent_sessions",
"current_mode",
"action_items",
"coaching_themes"
]
},
"communication": {
"text": {
"enabled": true
},
"voice": {
"enabled": false
}
},
"plugins": {
"entries": {}
},
"api": {
"enabled": true,
"port": 3000,
"cors": ["http://pipecat-agent:8080"]
}
}
```
---
## Step 7: Docker Stack Definition
### `docker-stack.yml`
```yaml
version: "3.8"
networks:
hobbs-overlay:
external: true
voicecoach-internal:
driver: overlay
internal: true # Not routable externally
volumes:
speechbrain-models:
driver: local
openclaw-voicecoach-data:
driver: local
livekit-data:
driver: local
secrets:
voicecoach_anthropic_key:
external: true
voicecoach_deepgram_key:
external: true
voicecoach_elevenlabs_key:
external: true
voicecoach_elevenlabs_voice_id:
external: true
voicecoach_openclaw_token:
external: true
voicecoach_livekit_api_key:
external: true
voicecoach_livekit_api_secret:
external: true
services:
# ──────────────────────────────────────────────────────────
# LiveKit WebRTC Server
# ──────────────────────────────────────────────────────────
livekit-server:
image: livekit/livekit-server:latest
command: --config /etc/livekit/livekit.yaml
ports:
- "7880:7880" # HTTP/WS API
- "7881:7881" # RTC over TCP
- "50000-60000:50000-60000/udp" # WebRTC media
volumes:
- ./livekit/livekit.yaml:/etc/livekit/livekit.yaml:ro
- livekit-data:/data
environment:
- LIVEKIT_KEYS_FILE=/run/secrets/voicecoach_livekit_api_key
secrets:
- voicecoach_livekit_api_key
- voicecoach_livekit_api_secret
networks:
- hobbs-overlay
- voicecoach-internal
deploy:
replicas: 1
placement:
constraints:
- node.role == manager # LiveKit benefits from stable placement
labels:
- "traefik.enable=true"
- "traefik.http.routers.livekit.rule=Host(`voice.hobbs.farm`)"
- "traefik.http.routers.livekit.entrypoints=websecure"
- "traefik.http.routers.livekit.tls.certresolver=cloudflare"
- "traefik.http.services.livekit.loadbalancer.server.port=7880"
- "traefik.http.routers.livekit.middlewares=livekit-websocket"
- "traefik.http.middlewares.livekit-websocket.headers.customrequestheaders.Upgrade=websocket"
restart_policy:
condition: on-failure
# ──────────────────────────────────────────────────────────
# SpeechBrain Emotion Analysis Microservice (INTERNAL ONLY)
# ──────────────────────────────────────────────────────────
speechbrain-service:
image: hobbs.farm/voicecoach/speechbrain:latest
build:
context: ./speechbrain-service
volumes:
- speechbrain-models:/app/models
environment:
- MODEL_DIR=/app/models
networks:
- voicecoach-internal
deploy:
replicas: 1
placement:
constraints:
- node.labels.gpu != true # Keep off GPU node unless configured
resources:
limits:
cpus: "2"
memory: "3G"
reservations:
memory: "2G"
restart_policy:
condition: on-failure
healthcheck:
test: ["CMD", "python", "-c", "import httpx; httpx.get('http://localhost:8001/health').raise_for_status()"]
interval: 30s
timeout: 10s
retries: 3
start_period: 90s # Model loading takes time
# ──────────────────────────────────────────────────────────
# Pipecat Voice Pipeline Agent
# ──────────────────────────────────────────────────────────
pipecat-agent:
image: hobbs.farm/voicecoach/pipecat:latest
build:
context: ./pipecat-agent
environment:
- LIVEKIT_URL=ws://livekit-server:7880
- SPEECHBRAIN_SERVICE_URL=http://speechbrain-service:8001
- OPENCLAW_URL=http://openclaw-voicecoach:3000
secrets:
- voicecoach_anthropic_key
- voicecoach_deepgram_key
- voicecoach_elevenlabs_key
- voicecoach_elevenlabs_voice_id
- voicecoach_livekit_api_key
- voicecoach_livekit_api_secret
- voicecoach_openclaw_token
networks:
- voicecoach-internal
- hobbs-overlay # Needs internet for Anthropic/Deepgram/ElevenLabs APIs
depends_on:
- livekit-server
- speechbrain-service
- openclaw-voicecoach
deploy:
replicas: 1
restart_policy:
condition: on-failure
delay: 10s
# ──────────────────────────────────────────────────────────
# OpenClaw VoiceCoach Agent
# ──────────────────────────────────────────────────────────
openclaw-voicecoach:
image: ghcr.io/openclaw/openclaw:latest
volumes:
- ./openclaw-agent/openclaw.json:/app/openclaw.json:ro
- ./openclaw-agent/system_prompt.md:/app/system_prompt.md:ro
- ./openclaw-agent/skills:/app/skills:ro
- openclaw-voicecoach-data:/app/data
secrets:
- voicecoach_anthropic_key
- voicecoach_openclaw_token
environment:
- OPENCLAW_CONFIG=/app/openclaw.json
- ANTHROPIC_API_KEY_FILE=/run/secrets/voicecoach_anthropic_key
networks:
- voicecoach-internal
- hobbs-overlay
deploy:
replicas: 1
restart_policy:
condition: on-failure
healthcheck:
test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
interval: 30s
timeout: 5s
retries: 3
# ──────────────────────────────────────────────────────────
# Web Client (phone interface)
# ──────────────────────────────────────────────────────────
voicecoach-web:
image: hobbs.farm/voicecoach/web:latest
build:
context: ./web-client
networks:
- hobbs-overlay
- voicecoach-internal
deploy:
replicas: 1
labels:
- "traefik.enable=true"
- "traefik.http.routers.voicecoach.rule=Host(`coach.hobbs.farm`)"
- "traefik.http.routers.voicecoach.entrypoints=websecure"
- "traefik.http.routers.voicecoach.tls.certresolver=cloudflare"
- "traefik.http.services.voicecoach.loadbalancer.server.port=80"
- "traefik.http.routers.voicecoach.middlewares=voicecoach-auth"
- "traefik.http.middlewares.voicecoach-auth.basicauth.users=jeremy:$$apr1$$REPLACEME"
restart_policy:
condition: on-failure
```
> **Note for Claude Code:** Replace `$$apr1$$REPLACEME` in the basicauth label with the output of:
> ```bash
> htpasswd -nb jeremy yourpassword | sed 's/\$/\$\$/g'
> ```
---
## Step 8: Makefile
### `Makefile`
```makefile
.PHONY: build deploy secrets test smoke e2e clean logs
# Build all images
build:
docker build -t hobbs.farm/voicecoach/speechbrain:latest ./speechbrain-service
docker build -t hobbs.farm/voicecoach/pipecat:latest ./pipecat-agent
docker build -t hobbs.farm/voicecoach/web:latest ./web-client
# Create secrets (run once)
secrets:
bash scripts/create-secrets.sh
# Deploy stack
deploy: build
docker stack deploy -c docker-stack.yml voicecoach
# Run unit tests (local, no Docker required)
test:
cd speechbrain-service && python -m pytest tests/ -v
cd pipecat-agent && python -m pytest tests/ -v
# Quick smoke test after deploy
smoke:
bash scripts/smoke-test.sh
# Full end-to-end test
e2e:
bash scripts/e2e-test.sh
# Follow all service logs
logs:
docker service logs voicecoach_pipecat-agent -f
# Remove stack
clean:
docker stack rm voicecoach
```
---
## Step 9: Test & Validation Scripts
### `scripts/smoke-test.sh`
```bash
#!/bin/bash
# Quick validation — run immediately after deploy
set -e
echo "=== VoiceCoach Smoke Tests ==="
# 1. Check all services running
echo ""
echo "→ Checking service replicas..."
docker service ls | grep voicecoach
FAILED=$(docker service ls | grep voicecoach | grep "0/" | wc -l)
if [ "$FAILED" -gt 0 ]; then
echo "❌ Some services have 0 replicas. Check: docker service ps voicecoach_<name>"
exit 1
fi
echo "✅ All services running"
# 2. LiveKit health
echo ""
echo "→ LiveKit health check..."
curl -sf https://voice.hobbs.farm/rtc/health > /dev/null && echo "✅ LiveKit responding" || echo "❌ LiveKit not responding"
# 3. SpeechBrain health (via internal overlay — run from swarm manager)
echo ""
echo "→ SpeechBrain health check (internal)..."
# Run from a container on the voicecoach-internal network
docker run --rm --network voicecoach_voicecoach-internal \
curlimages/curl:latest \
curl -sf http://speechbrain-service:8001/health && echo "✅ SpeechBrain responding" || echo "❌ SpeechBrain not responding"
# 4. Web client
echo ""
echo "→ Web client check..."
curl -sf -u jeremy:yourpassword https://coach.hobbs.farm > /dev/null && echo "✅ Web client responding" || echo "❌ Web client not responding"
# 5. OpenClaw health
echo ""
echo "→ OpenClaw health check (internal)..."
docker run --rm --network voicecoach_voicecoach-internal \
curlimages/curl:latest \
curl -sf http://openclaw-voicecoach:3000/health && echo "✅ OpenClaw responding" || echo "❌ OpenClaw not responding"
echo ""
echo "=== Smoke tests complete ==="
```
### `scripts/e2e-test.sh`
```bash
#!/bin/bash
# Full end-to-end test using a pre-recorded test WAV
# Requires: livekit-cli installed (go install github.com/livekit/livekit-cli/cmd/lk@latest)
echo "=== VoiceCoach End-to-End Test ==="
echo "This test joins the LiveKit room as a test participant and"
echo "sends a pre-recorded audio clip, then checks for TTS output."
LK_KEY=$(docker secret inspect voicecoach_livekit_api_key --format '{{.Spec.Name}}' 2>/dev/null || echo "not-found")
echo ""
echo "→ Downloading test audio..."
curl -sL "https://www2.cs.uic.edu/~i101/SoundFiles/StarWars60.wav" -o /tmp/test_audio.wav
echo ""
echo "→ Joining LiveKit room and sending test audio..."
echo " (Manual step: open https://coach.hobbs.farm on phone and verify you hear a response)"
echo ""
echo "→ Alternatively, use livekit-cli:"
echo " lk room join --url wss://voice.hobbs.farm --api-key \$KEY --api-secret \$SECRET voicecoach"
echo ""
echo "=== E2E test requires manual verification at this time ==="
echo "→ Automated e2e test with audio injection coming in v1.1"
```
### `tests/test_latency.sh`
```bash
#!/bin/bash
# Measure SpeechBrain analysis latency
echo "=== Latency Test: SpeechBrain Analysis ==="
# Download a 2-second speech sample
curl -sL "https://www2.cs.uic.edu/~i101/SoundFiles/preamble.wav" -o /tmp/latency_test.wav
# Test via internal network
START=$(date +%s%3N)
docker run --rm --network voicecoach_voicecoach-internal \
-v /tmp/latency_test.wav:/test.wav \
curlimages/curl:latest \
curl -sf -X POST http://speechbrain-service:8001/analyze \
-F "audio=@/test.wav" > /tmp/latency_result.json
END=$(date +%s%3N)
ELAPSED=$((END - START))
echo "Analysis latency: ${ELAPSED}ms"
if [ "$ELAPSED" -lt 3000 ]; then
echo "✅ Within 3s budget"
else
echo "⚠️ Exceeds 3s budget — consider GPU or lighter model"
fi
echo ""
cat /tmp/latency_result.json | python3 -m json.tool
```
---
## Step 10: Deploy Order
Claude Code should execute in this exact order:
```bash
# 1. Clone/create repository structure
# (Claude Code: create all files from this document)
# 2. Create Docker Swarm secrets (ONCE)
make secrets
# 3. Confirm overlay network
docker network ls | grep hobbs-overlay || docker network create --driver overlay --attachable hobbs-overlay
# 4. Build images
make build
# 5. Deploy stack
make deploy
# 6. Monitor startup (SpeechBrain takes ~90s to load model)
watch docker service ls
# 7. Run smoke tests
make smoke
# 8. Run unit tests
make test
# 9. Open coach.hobbs.farm on phone and test voice
```
---
## Interface Change Summary
**What changes from your current Matrix setup:**
| Capability | Before | After |
|---|---|---|
| Real-time voice with agents | ❌ Not supported | ✅ coach.hobbs.farm (phone browser) |
| Text chat with Pluto/agents | ✅ Matrix stays | ✅ Matrix stays (unchanged) |
| Async task dispatch | ✅ Matrix/Dispatch | ✅ Matrix/Dispatch stays |
| Vocal analysis coaching | ❌ Not possible | ✅ Full pipeline |
| Session memory | ✅ OpenClaw memory | ✅ OpenClaw memory (enhanced) |
**Matrix is NOT replaced.** It remains your primary async interface for Pluto and all text-based agent interaction. VoiceCoach is additive — a dedicated real-time voice interface for commute sessions only.
---
## Rollback
```bash
# Remove the voice coach stack entirely (OpenClaw memory data preserved in volume)
docker stack rm voicecoach
# Remove secrets if rotating
docker secret rm voicecoach_anthropic_key voicecoach_deepgram_key \
voicecoach_elevenlabs_key voicecoach_elevenlabs_voice_id \
voicecoach_openclaw_token voicecoach_livekit_api_key voicecoach_livekit_api_secret
```
---
## Known Issues & Troubleshooting
**SpeechBrain takes too long on CPU:**
- First inference is always slower (model warmup). Subsequent calls are faster.
- If consistently > 3s: add a lighter model path using librosa-only analysis (no SpeechBrain) as fallback.
**WebRTC media not flowing (hear nothing):**
- Almost certainly firewall/NAT. Confirm UDP 50000-60000 is open at router.
- LiveKit logs: `docker service logs voicecoach_livekit-server -f`
**Deepgram returning no transcripts:**
- Check API key secret. Deepgram errors appear in pipecat-agent logs.
- Try `endpointing=1000` if utterances are being cut off.
**ElevenLabs high latency:**
- Switch to `eleven_turbo_v2_5` model (already set). If still slow, use `edge-tts` locally.
- Set `optimize_streaming_latency=4` (already set — maximum optimization).
**SpeechBrain emotion always "neutral":**
- Normal for the IEMOCAP model on non-actor speech. Acoustic features (pitch, energy, rate) are more reliable than the emotion label for real speech. Rely on those for coaching signals.
Reference in New Issue View Git Blame Copy Permalink