# VoiceCoach — Homelab Deployment Instructions ### Pipecat + SpeechBrain + Claude + LiveKit + OpenClaw **Target: hobbs.farm Docker Swarm | Author: Claude | For use with Claude Code** --- ## Requirements Iteration Log > Claude Code: Do NOT skip this section. It documents why decisions were made and will save you from dead ends. ### Iteration 1 — Core Functional Requirements (Raw) Initial capture of what this system must do: - Receive real-time voice from Jeremy's phone while driving - Transcribe speech to text - Analyze vocal characteristics (pitch, energy, emotion, speaking rate) - Pass both transcript AND vocal metadata to Claude - Claude responds with coaching feedback delivered as spoken audio - Session memory persists across commutes - Self-hosted on hobbs.farm infrastructure **Gap identified:** No interface specified. Matrix is async/text-focused — not suitable for real-time bidirectional audio streaming. Need a WebRTC solution. --- ### Iteration 2 — Infrastructure Constraint Pass Applying Jeremy's known infrastructure to the requirements: - **Must deploy as Docker Swarm stack** — nodes: hf-pdocker-01, hf-pdocker-02, bart - **Must use Traefik** for SSL termination with existing wildcard `*.hobbs.farm` cert via Cloudflare DNS challenge - **Must integrate with OpenClaw** — new agent instance, not a standalone app - **Must store conversation memory** in OpenClaw's memory system (compatible with Pluto) - **Interface decision:** Replace Matrix voice with **LiveKit** (self-hosted WebRTC) at `voice.hobbs.farm`, web client at `coach.hobbs.farm`. Matrix retained for text/async agent interaction only. - **SpeechBrain must run as its own microservice** — not embedded in pipeline — so it can be reused and scaled independently - **No GPU assumption for initial deploy** — SpeechBrain runs on CPU. GPU path documented but optional. **Gap identified:** Need to define the exact data contract between SpeechBrain service and Pipecat pipeline. What fields get passed to Claude and in what format. --- ### Iteration 3 — SpeechBrain & Emotion Pipeline Detail Defining the data contract and model choices: **SpeechBrain Model:** `speechbrain/emotion-recognition-wav2vec2-IEMOCAP` - Classifies: neutral, happy, sad, angry, frustrated - Input: 16kHz mono WAV, minimum ~1 second - Output: emotion label + per-class confidence scores **Additional acoustic features via librosa:** - `pitch_mean_hz` — fundamental frequency average - `pitch_variance` — pitch stability (high variance = animated/stressed) - `energy_rms` — loudness/intensity - `speaking_rate_syllables_per_sec` — estimated via onset detection - `voiced_fraction` — ratio of voiced to unvoiced frames (low = hesitant/uncertain) **Prompt injection format (into every Claude turn):** ``` [VOCAL_STATE emotion=frustrated confidence=0.82 pitch=215hz pitch_var=high energy=0.71 rate=4.8sps voiced=0.91] User said: "I just cannot get the team to listen to me in standups" ``` **Claude system prompt must instruct:** - Interpret `[VOCAL_STATE]` tags as paralinguistic context - Distinguish between what the user says and how they sound saying it - Provide coaching on both content and delivery when relevant - Never mention the raw numbers — translate to natural coaching language **Gap identified:** How does Pipecat know when an utterance is complete to trigger SpeechBrain? Need end-of-turn detection strategy. Deepgram handles this via `is_final` events in streaming mode. --- ### Iteration 4 — Testing, Validation, Security Defining what "working" means before writing code: **Unit Tests:** - `test_speechbrain_service.py` — POST audio file, assert emotion field present, confidence > 0, latency < 2s - `test_pipecat_pipeline.py` — mock Deepgram + SpeechBrain, assert Claude receives correctly formatted prompt - `test_openclaw_agent.py` — assert agent responds and memory is written after session **Integration Tests:** - `test_e2e_pipeline.sh` — send WAV file to LiveKit room via livekit-cli, assert TTS audio received back - `test_latency.sh` — measure total roundtrip from audio-in to audio-out, assert < 4s **Docker Health Checks:** - SpeechBrain service: `GET /health` returns 200, model loaded - Pipecat agent: process alive check - LiveKit: `GET /rtc/health` **Validation Steps (run before marking deploy complete):** 1. `docker service ls` — all 6 services showing replicas as desired 2. `curl https://voice.hobbs.farm/rtc/health` — LiveKit responding 3. `curl https://coach.hobbs.farm` — web client loads 4. SpeechBrain smoke test with provided sample audio 5. Full end-to-end: open coach.hobbs.farm on phone, say "test", hear Claude respond **Security:** - LiveKit API key/secret stored as Docker Swarm secrets (not env vars) - Deepgram API key as Swarm secret - Anthropic API key as Swarm secret - ElevenLabs API key as Swarm secret - coach.hobbs.farm protected by basic auth via Traefik middleware (single user) - SpeechBrain service NOT exposed externally — internal Swarm overlay network only - OpenClaw agent token as Swarm secret **Gap identified:** Need to address baseline calibration — SpeechBrain trained on IEMOCAP actors, not Jeremy's voice. First session should establish Jeremy's neutral baseline so relative changes are meaningful. --- ### Iteration 5 — Memory, Continuity & Calibration Addressing the coaching quality requirements: **Baseline Calibration (first session only):** - First 60 seconds of first session: Claude prompts Jeremy to read a neutral calibration script - SpeechBrain readings stored in OpenClaw memory as `vocal_baseline` profile - Subsequent sessions: all readings compared to baseline, delta reported to Claude - Format: `[VOCAL_STATE emotion=frustrated delta_from_baseline: pitch+18hz energy+0.12]` **Session Memory Structure in OpenClaw:** ```json { "session_id": "2026-04-01-commute-am", "duration_minutes": 22, "topics_discussed": ["team communication", "standup facilitation"], "vocal_trends": { "avg_stress_level": 0.71, "peak_frustration_at": "12:34", "improvement_noted": false }, "coaching_notes": "Jeremy shows elevated pitch when discussing team dynamics", "action_items": ["Practice lower speaking rate in next standup"] } ``` **Cross-session coaching continuity:** - OpenClaw system prompt includes last 3 session summaries - Claude tracks whether previous action items were attempted - Vocal improvement over time is explicitly tracked and acknowledged **Commute-optimized coaching modes (user-selectable via wake phrase):** - `"coaching mode"` — deep conversation, feedback on every utterance - `"vent mode"` — Claude listens, reflects back, minimal interruption - `"practice mode"` — Claude plays a difficult stakeholder, Jeremy practices - `"briefing mode"` — Claude summarizes pending tasks from Pluto/OpenClaw memory **Gap identified:** Need to define how mode switching works mid-conversation without typing. Answer: wake phrases detected at STT layer, routed before Claude prompt. --- ### Iteration 6 — Latency Budget, Fallbacks & Observability Ensuring the system is actually usable in a moving truck: **Latency Budget (target < 3.5s total roundtrip):** - Deepgram streaming STT end-of-turn detection: ~200ms - SpeechBrain analysis (async, overlaps with STT): ~800ms on CPU - Claude API call: ~1200ms - ElevenLabs TTS first audio chunk: ~400ms - LiveKit transmission: ~100ms - **Total: ~2.7s** — within target **Fallbacks:** - If SpeechBrain service unreachable: pipeline continues without vocal metadata, logs warning - If Deepgram fails: fallback to Whisper local STT (slower but self-contained) - If ElevenLabs fails: fallback to edge-tts (local, lower quality) - If Claude API fails: return "I'm having trouble connecting, please try again" **Observability:** - All services log to Docker's logging driver - Pipecat pipeline emits structured JSON logs per utterance including latencies - SpeechBrain service logs emotion distribution per session (useful for trend analysis) - Consider forwarding to existing monitoring stack if Prometheus/Grafana present **Mobile network resilience:** - LiveKit handles packet loss via WebRTC NACK/FEC automatically - Configure Deepgram with `endpointing=500` (500ms silence = end of utterance) — more forgiving for noisy truck cab - Push-to-talk fallback mode available via coach.hobbs.farm UI (hold button while speaking) --- ### Iteration 7 — Claude Code Readiness Pass Final pass ensuring Claude Code can execute this without ambiguity: **Explicit file structure defined** — Claude Code should create exactly these files **All environment variables listed** with descriptions and example values **Each service has its own section** with Dockerfile, config, and test **Deploy order specified** — secrets → network → redis → speechbrain → livekit → pipecat → openclaw → web client **Rollback procedure documented** **Every `docker stack deploy` command is copy-pasteable** **Tests are runnable with a single command after deploy** --- ## System Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ PHONE (in truck) │ │ coach.hobbs.farm (Chrome/Safari browser) │ │ LiveKit JS SDK — WebRTC audio stream │ └──────────────────────┬──────────────────────────────────┘ │ WebRTC (DTLS/SRTP) ▼ ┌─────────────────────────────────────────────────────────┐ │ LiveKit Server (voice.hobbs.farm) │ │ WebRTC SFU — routes audio frames │ └──────────┬───────────────────────────────────┬──────────┘ │ LiveKit SDK (internal) │ LiveKit SDK ▼ ▼ ┌──────────────────────┐ ┌──────────────────────┐ │ Pipecat Pipeline │ │ (future: recording) │ │ Agent Process │ └──────────────────────┘ │ │ │ ┌────────────────┐ │ │ │ Deepgram STT │ │ ← streaming audio chunks │ │ (streaming) │ │ │ └───────┬────────┘ │ │ │ transcript │ │ ▼ │ │ ┌────────────────┐ │ │ │ SpeechBrain │──┼──→ POST /analyze (internal overlay network) │ │ Emotion Client │ │ ← {emotion, pitch, energy, rate, voiced} │ └───────┬────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────┐ │ │ │ Claude API │ │ ← formatted prompt with [VOCAL_STATE] │ │ claude-sonnet │ │ → coaching response text │ └───────┬────────┘ │ │ │ │ │ ▼ │ │ ┌────────────────┐ │ │ │ ElevenLabs TTS │ │ │ │ (streaming) │ │ │ └───────┬────────┘ │ │ │ audio │ └──────────┼────────────┘ │ audio frames back to LiveKit ▼ Phone speaker ┌──────────────────────────────────────────────┐ │ SpeechBrain Microservice │ │ (internal network only) │ │ FastAPI + wav2vec2-IEMOCAP + librosa │ │ POST /analyze → emotion + prosody JSON │ │ GET /health → 200 OK │ │ GET /calibrate → session baseline │ └──────────────────────────────────────────────┘ ┌──────────────────────────────────────────────┐ │ OpenClaw VoiceCoach Agent │ │ - Manages session lifecycle │ │ - Writes memory after each session │ │ - Exposes /session/start, /session/end │ │ - Integrates with Pluto's memory store │ └──────────────────────────────────────────────┘ ``` --- ## Repository Structure Claude Code should create this exact directory structure: ``` voicecoach/ ├── docker-stack.yml # Main Swarm stack definition ├── .env.example # Template for secrets (never commit .env) ├── Makefile # Convenience targets │ ├── speechbrain-service/ │ ├── Dockerfile │ ├── requirements.txt │ ├── main.py # FastAPI app │ ├── analyzer.py # SpeechBrain + librosa logic │ ├── models/ # Model cache (mount as volume) │ └── tests/ │ ├── test_analyzer.py │ └── fixtures/ │ ├── neutral_sample.wav │ ├── angry_sample.wav │ └── stressed_sample.wav │ ├── pipecat-agent/ │ ├── Dockerfile │ ├── requirements.txt │ ├── main.py # Pipecat pipeline entry │ ├── pipeline.py # Pipeline assembly │ ├── prompt_builder.py # Builds Claude prompts with vocal state │ ├── mode_detector.py # Wake phrase detection │ ├── openclaw_client.py # Writes session memory to OpenClaw │ └── tests/ │ ├── test_pipeline.py │ └── test_prompt_builder.py │ ├── openclaw-agent/ │ ├── openclaw.json # OpenClaw agent config │ ├── system_prompt.md # VoiceCoach persona & instructions │ └── skills/ │ └── vocal_memory.md # Skill for reading/writing vocal baselines │ ├── web-client/ │ ├── Dockerfile # nginx serving static files │ ├── index.html │ ├── app.js # LiveKit JS SDK integration │ └── style.css │ ├── livekit/ │ └── livekit.yaml # LiveKit server config │ ├── scripts/ │ ├── create-secrets.sh # Creates all Docker Swarm secrets │ ├── smoke-test.sh # Quick validation after deploy │ ├── e2e-test.sh # Full end-to-end test │ └── calibrate.sh # Trigger baseline calibration session │ └── tests/ └── test_latency.sh # Roundtrip latency measurement ``` --- ## Step 0: Prerequisites Checklist Before Claude Code runs any deployment commands, verify: ```bash # 1. Confirm Swarm manager access docker node ls # 2. Confirm Traefik is running with wildcard cert docker service ls | grep traefik # 3. Confirm overlay network exists (or create it) docker network ls | grep hobbs-overlay # If missing: docker network create --driver overlay --attachable hobbs-overlay # 4. Confirm required external API keys are available # - Anthropic API key (claude-sonnet-4-6) # - Deepgram API key (https://console.deepgram.com) # - ElevenLabs API key (https://elevenlabs.io) # - LiveKit will use self-generated key pair (no external account needed) # 5. Confirm ports available # - 7880/tcp LiveKit HTTPS API # - 7881/tcp LiveKit RTC over TCP fallback # - 50000-60000/udp LiveKit WebRTC media (must be open on firewall/router) ``` **IMPORTANT:** If ports 50000-60000 UDP are not open at the router level, WebRTC media will not flow. This is the most common failure point. Confirm with: ```bash # From external network (not homelab): nmap -sU -p 50000-50010 ``` --- ## Step 1: Create Docker Swarm Secrets **File: `scripts/create-secrets.sh`** ```bash #!/bin/bash # Run this ONCE on the Swarm manager node. # Have your API keys ready before running. set -e echo "Creating Docker Swarm secrets for VoiceCoach..." read -s -p "Anthropic API Key: " ANTHROPIC_KEY && echo read -s -p "Deepgram API Key: " DEEPGRAM_KEY && echo read -s -p "ElevenLabs API Key: " ELEVENLABS_KEY && echo read -s -p "ElevenLabs Voice ID: " ELEVENLABS_VOICE_ID && echo read -s -p "OpenClaw Agent Token: " OPENCLAW_TOKEN && echo # Generate LiveKit key pair LIVEKIT_API_KEY="voicecoach-$(openssl rand -hex 4)" LIVEKIT_API_SECRET=$(openssl rand -base64 32) echo "$ANTHROPIC_KEY" | docker secret create voicecoach_anthropic_key - echo "$DEEPGRAM_KEY" | docker secret create voicecoach_deepgram_key - echo "$ELEVENLABS_KEY" | docker secret create voicecoach_elevenlabs_key - echo "$ELEVENLABS_VOICE_ID" | docker secret create voicecoach_elevenlabs_voice_id - echo "$OPENCLAW_TOKEN" | docker secret create voicecoach_openclaw_token - echo "$LIVEKIT_API_KEY" | docker secret create voicecoach_livekit_api_key - echo "$LIVEKIT_API_SECRET" | docker secret create voicecoach_livekit_api_secret - echo "" echo "✅ Secrets created. LiveKit credentials:" echo " API Key: $LIVEKIT_API_KEY" echo " API Secret: $LIVEKIT_API_SECRET" echo "" echo "⚠️ Save these LiveKit credentials — they are stored in Swarm secrets" echo " and cannot be retrieved again." ``` --- ## Step 2: SpeechBrain Microservice ### `speechbrain-service/requirements.txt` ``` fastapi==0.115.0 uvicorn[standard]==0.30.0 speechbrain==1.0.0 torch==2.3.0 torchaudio==2.3.0 librosa==0.10.2 soundfile==0.12.1 numpy==1.26.4 httpx==0.27.0 python-multipart==0.0.9 ``` ### `speechbrain-service/analyzer.py` ```python """ SpeechBrain + librosa vocal analysis. Performs emotion classification and acoustic feature extraction from audio. """ import io import logging import numpy as np import soundfile as sf import librosa from speechbrain.inference.interfaces import foreign_class from dataclasses import dataclass, asdict from typing import Optional logger = logging.getLogger(__name__) SAMPLE_RATE = 16000 MIN_DURATION_SECONDS = 0.5 # Reject clips shorter than this @dataclass class VocalAnalysis: emotion: str emotion_confidence: float emotion_scores: dict # All class probabilities pitch_mean_hz: Optional[float] pitch_variance: Optional[float] energy_rms: float speaking_rate_approx: float # Onset events per second (proxy for syllable rate) voiced_fraction: float # 0-1, fraction of voiced frames duration_seconds: float analysis_error: Optional[str] = None class VocalAnalyzer: """Loads models once at startup and provides analysis per audio chunk.""" def __init__(self, model_dir: str = "/app/models"): logger.info("Loading SpeechBrain emotion model...") self.classifier = foreign_class( source="speechbrain/emotion-recognition-wav2vec2-IEMOCAP", pymodule_file="custom_interface.py", classname="CustomEncoderWav2vec2Classifier", savedir=model_dir, run_opts={"device": "cpu"}, # Change to "cuda" if GPU available ) logger.info("SpeechBrain model loaded successfully.") def analyze(self, audio_bytes: bytes, content_type: str = "audio/wav") -> VocalAnalysis: """ Analyze audio bytes and return vocal features. Args: audio_bytes: Raw audio file bytes (WAV, FLAC, or OGG) content_type: MIME type hint Returns: VocalAnalysis dataclass """ try: # Load audio audio_buf = io.BytesIO(audio_bytes) audio, sr = sf.read(audio_buf, dtype="float32") # Ensure mono if audio.ndim > 1: audio = audio.mean(axis=1) # Resample to 16kHz if needed if sr != SAMPLE_RATE: audio = librosa.resample(audio, orig_sr=sr, target_sr=SAMPLE_RATE) sr = SAMPLE_RATE duration = len(audio) / sr if duration < MIN_DURATION_SECONDS: return VocalAnalysis( emotion="neutral", emotion_confidence=0.0, emotion_scores={}, pitch_mean_hz=None, pitch_variance=None, energy_rms=0.0, speaking_rate_approx=0.0, voiced_fraction=0.0, duration_seconds=duration, analysis_error=f"Audio too short: {duration:.2f}s < {MIN_DURATION_SECONDS}s minimum", ) # --- SpeechBrain emotion classification --- import tempfile, os with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp: sf.write(tmp.name, audio, sr) tmp_path = tmp.name try: out_prob, score, index, text_lab = self.classifier.classify_file(tmp_path) emotion = text_lab[0] if isinstance(text_lab, list) else text_lab confidence = float(score[0]) if hasattr(score, "__iter__") else float(score) # Build score dict from probability tensor labels = ["neu", "hap", "sad", "ang", "fru"] # IEMOCAP classes prob_list = out_prob[0].tolist() if hasattr(out_prob[0], "tolist") else list(out_prob[0]) emotion_scores = {l: round(p, 4) for l, p in zip(labels, prob_list)} finally: os.unlink(tmp_path) # --- Librosa acoustic features --- # Pitch via pyin (more accurate than yin for speech) f0, voiced_flag, _ = librosa.pyin( audio, fmin=librosa.note_to_hz("C2"), fmax=librosa.note_to_hz("C7"), sr=sr, ) voiced_f0 = f0[voiced_flag] if voiced_flag is not None else np.array([]) pitch_mean = float(np.mean(voiced_f0)) if len(voiced_f0) > 0 else None pitch_var = float(np.var(voiced_f0)) if len(voiced_f0) > 1 else None voiced_fraction = float(np.sum(voiced_flag) / len(voiced_flag)) if voiced_flag is not None else 0.0 # Energy rms = librosa.feature.rms(y=audio) energy_rms = float(np.mean(rms)) # Speaking rate proxy: onset strength events per second onset_frames = librosa.onset.onset_detect(y=audio, sr=sr) speaking_rate = len(onset_frames) / duration if duration > 0 else 0.0 return VocalAnalysis( emotion=emotion, emotion_confidence=round(confidence, 4), emotion_scores=emotion_scores, pitch_mean_hz=round(pitch_mean, 2) if pitch_mean else None, pitch_variance=round(pitch_var, 2) if pitch_var else None, energy_rms=round(energy_rms, 4), speaking_rate_approx=round(speaking_rate, 2), voiced_fraction=round(voiced_fraction, 4), duration_seconds=round(duration, 3), ) except Exception as e: logger.exception(f"Analysis failed: {e}") return VocalAnalysis( emotion="neutral", emotion_confidence=0.0, emotion_scores={}, pitch_mean_hz=None, pitch_variance=None, energy_rms=0.0, speaking_rate_approx=0.0, voiced_fraction=0.0, duration_seconds=0.0, analysis_error=str(e), ) ``` ### `speechbrain-service/main.py` ```python """ FastAPI service wrapping VocalAnalyzer. Internal-only — not exposed via Traefik. """ import logging import os from contextlib import asynccontextmanager from fastapi import FastAPI, UploadFile, File, HTTPException from fastapi.responses import JSONResponse from analyzer import VocalAnalyzer, asdict logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) MODEL_DIR = os.getenv("MODEL_DIR", "/app/models") analyzer: VocalAnalyzer = None @asynccontextmanager async def lifespan(app: FastAPI): global analyzer logger.info("Starting VocalAnalyzer...") analyzer = VocalAnalyzer(model_dir=MODEL_DIR) logger.info("VocalAnalyzer ready.") yield logger.info("Shutting down.") app = FastAPI(title="SpeechBrain Vocal Analysis Service", lifespan=lifespan) @app.get("/health") async def health(): if analyzer is None: raise HTTPException(status_code=503, detail="Model not loaded") return {"status": "ok", "model": "emotion-recognition-wav2vec2-IEMOCAP"} @app.post("/analyze") async def analyze(audio: UploadFile = File(...)): """ Analyze vocal characteristics of an audio clip. Accepts: WAV, FLAC, OGG audio files. Returns: Emotion classification + acoustic features. """ if analyzer is None: raise HTTPException(status_code=503, detail="Model not loaded") content = await audio.read() if len(content) == 0: raise HTTPException(status_code=400, detail="Empty audio file") result = analyzer.analyze(content, content_type=audio.content_type or "audio/wav") return JSONResponse(content=asdict(result)) @app.post("/calibrate") async def calibrate(audio: UploadFile = File(...)): """ Analyze audio and return it labeled as baseline calibration. Same as /analyze but semantically marks this as a neutral reference. """ result_response = await analyze(audio) data = result_response.body import json result_dict = json.loads(data) result_dict["is_baseline"] = True return JSONResponse(content=result_dict) ``` ### `speechbrain-service/Dockerfile` ```dockerfile FROM python:3.11-slim # System deps for librosa/soundfile RUN apt-get update && apt-get install -y \ libsndfile1 \ ffmpeg \ git \ && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Pre-download the SpeechBrain model at build time to avoid cold start # This bakes the model into the image (~400MB) RUN python -c " from speechbrain.inference.interfaces import foreign_class foreign_class( source='speechbrain/emotion-recognition-wav2vec2-IEMOCAP', pymodule_file='custom_interface.py', classname='CustomEncoderWav2vec2Classifier', savedir='/app/models', run_opts={'device': 'cpu'}, ) print('Model downloaded successfully.') " COPY . . EXPOSE 8001 HEALTHCHECK --interval=30s --timeout=10s --start-period=60s \ CMD python -c "import httpx; httpx.get('http://localhost:8001/health').raise_for_status()" CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001", "--workers", "1"] ``` ### `speechbrain-service/tests/test_analyzer.py` ```python """ Tests for the SpeechBrain vocal analyzer. Requires test WAV fixtures to be present. """ import os import pytest import numpy as np import soundfile as sf from pathlib import Path FIXTURES = Path(__file__).parent / "fixtures" def create_sine_wav(path: Path, frequency: float = 440.0, duration: float = 2.0, sr: int = 16000): """Generate a sine wave WAV file for testing.""" t = np.linspace(0, duration, int(sr * duration)) audio = 0.5 * np.sin(2 * np.pi * frequency * t).astype(np.float32) sf.write(str(path), audio, sr) @pytest.fixture(scope="module", autouse=True) def create_fixtures(): FIXTURES.mkdir(exist_ok=True) create_sine_wav(FIXTURES / "neutral_sample.wav", frequency=200, duration=2.0) create_sine_wav(FIXTURES / "short_sample.wav", frequency=200, duration=0.2) def test_analyzer_loads(): """Analyzer must load without error.""" from analyzer import VocalAnalyzer a = VocalAnalyzer() assert a is not None def test_analyze_returns_emotion(): from analyzer import VocalAnalyzer a = VocalAnalyzer() audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes() result = a.analyze(audio_bytes) assert result.emotion in ["neu", "hap", "sad", "ang", "fru"] assert 0.0 <= result.emotion_confidence <= 1.0 def test_analyze_returns_acoustic_features(): from analyzer import VocalAnalyzer a = VocalAnalyzer() audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes() result = a.analyze(audio_bytes) assert result.energy_rms >= 0.0 assert result.duration_seconds > 0.0 assert 0.0 <= result.voiced_fraction <= 1.0 def test_short_audio_handled_gracefully(): from analyzer import VocalAnalyzer a = VocalAnalyzer() audio_bytes = (FIXTURES / "short_sample.wav").read_bytes() result = a.analyze(audio_bytes) assert result.analysis_error is not None assert "too short" in result.analysis_error.lower() def test_analyze_latency(): """Analysis should complete in under 3 seconds on CPU.""" import time from analyzer import VocalAnalyzer a = VocalAnalyzer() audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes() start = time.time() a.analyze(audio_bytes) elapsed = time.time() - start assert elapsed < 3.0, f"Analysis took {elapsed:.2f}s — exceeds 3s budget" ``` --- ## Step 3: Pipecat Voice Pipeline Agent ### `pipecat-agent/requirements.txt` ``` pipecat-ai[deepgram,elevenlabs,livekit,silero]==0.0.48 anthropic==0.34.0 httpx==0.27.0 python-dotenv==1.0.1 structlog==24.4.0 ``` ### `pipecat-agent/prompt_builder.py` ```python """ Builds Claude prompts with embedded vocal state metadata. """ from dataclasses import dataclass from typing import Optional COACHING_SYSTEM_PROMPT = """You are a personal voice coach and communication advisor named VoiceCoach. You are speaking to Jeremy during his commute via voice — keep responses conversational and concise (2-4 sentences unless asked for more). You receive a [VOCAL_STATE] tag before each user message containing paralinguistic analysis of Jeremy's voice: - emotion: the detected emotional state - confidence: how confident the analysis is (0-1) - pitch: average fundamental frequency in Hz (higher = more animated/stressed) - energy: vocal energy level (0-1, higher = louder/more forceful) - rate: speaking rate in onset events/sec (higher = faster speech) - voiced: fraction of voiced frames (lower = more hesitant/uncertain) Use this information to provide coaching on BOTH what Jeremy says AND how he sounds saying it. Important: Never quote the raw numbers. Translate them into natural observations. Example: "I notice some tension in your voice..." not "Your pitch was 215hz..." {baseline_context} {session_history} Available coaching modes (Jeremy can switch by saying the phrase): - "coaching mode" (default) — active feedback on content and delivery - "vent mode" — listen and reflect, minimal advice - "practice mode" — roleplay a difficult stakeholder - "briefing mode" — summarize Jeremy's tasks and context from memory Current mode: {current_mode} """ @dataclass class VocalState: emotion: str confidence: float pitch_mean_hz: Optional[float] energy_rms: float speaking_rate_approx: float voiced_fraction: float def format_vocal_state(state: VocalState) -> str: """Format vocal state as the [VOCAL_STATE] tag injected before each user message.""" pitch_str = f"{state.pitch_mean_hz:.0f}hz" if state.pitch_mean_hz else "unknown" return ( f"[VOCAL_STATE " f"emotion={state.emotion} " f"confidence={state.confidence:.2f} " f"pitch={pitch_str} " f"energy={state.energy_rms:.2f} " f"rate={state.speaking_rate_approx:.1f}ops " f"voiced={state.voiced_fraction:.2f}]" ) def build_user_message(transcript: str, vocal_state: Optional[VocalState]) -> str: """Combines transcript with vocal state tag.""" if vocal_state and vocal_state.confidence > 0.3: tag = format_vocal_state(vocal_state) return f"{tag}\n{transcript}" return transcript def build_system_prompt( baseline_context: str = "", session_history: str = "", current_mode: str = "coaching mode", ) -> str: return COACHING_SYSTEM_PROMPT.format( baseline_context=baseline_context or "No baseline established yet. First session: ask Jeremy to read a short neutral calibration script.", session_history=session_history or "No previous session history.", current_mode=current_mode, ) ``` ### `pipecat-agent/pipeline.py` ```python """ Main Pipecat pipeline assembly. Connects LiveKit → Deepgram → SpeechBrain → Claude → ElevenLabs → LiveKit. """ import asyncio import os import logging import httpx from typing import Optional from pipecat.frames.frames import ( AudioRawFrame, TranscriptionFrame, TextFrame, EndFrame ) from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.services.anthropic import AnthropicLLMService from pipecat.services.deepgram import DeepgramSTTService, LiveOptions from pipecat.services.elevenlabs import ElevenLabsTTSService from pipecat.transports.services.livekit import LiveKitTransport, LiveKitParams from pipecat.processors.frame_processor import FrameProcessor, FrameDirection from pipecat.vad.silero import SileroVADAnalyzer from prompt_builder import build_system_prompt, build_user_message, VocalState logger = logging.getLogger(__name__) SPEECHBRAIN_URL = os.getenv("SPEECHBRAIN_SERVICE_URL", "http://speechbrain-service:8001") ANTHROPIC_MODEL = "claude-sonnet-4-6" class VocalStateInjector(FrameProcessor): """ Intercepts TranscriptionFrames, fetches vocal analysis from SpeechBrain, and enriches the frame before passing to the LLM context. Audio buffer is maintained separately from the STT pipeline — SpeechBrain receives a copy of the raw audio for each utterance. """ def __init__(self, **kwargs): super().__init__(**kwargs) self._current_audio: list[bytes] = [] self._http_client = httpx.AsyncClient(timeout=3.0) async def process_frame(self, frame, direction: FrameDirection): await super().process_frame(frame, direction) if isinstance(frame, AudioRawFrame): # Buffer audio for the current utterance self._current_audio.append(frame.audio) elif isinstance(frame, TranscriptionFrame): # Utterance complete — analyze buffered audio vocal_state = await self._analyze_buffered_audio() self._current_audio = [] # Reset buffer # Inject vocal state into the transcript enriched_text = build_user_message(frame.text, vocal_state) enriched_frame = TranscriptionFrame( text=enriched_text, user_id=frame.user_id, timestamp=frame.timestamp, ) await self.push_frame(enriched_frame, direction) else: await self.push_frame(frame, direction) async def _analyze_buffered_audio(self) -> Optional[VocalState]: if not self._current_audio: return None try: import io, wave audio_bytes = b"".join(self._current_audio) # Wrap raw PCM in WAV container for SpeechBrain buf = io.BytesIO() with wave.open(buf, "wb") as wf: wf.setnchannels(1) wf.setsampwidth(2) # 16-bit wf.setframerate(16000) wf.writeframes(audio_bytes) wav_bytes = buf.getvalue() response = await self._http_client.post( f"{SPEECHBRAIN_URL}/analyze", files={"audio": ("utterance.wav", wav_bytes, "audio/wav")}, ) response.raise_for_status() data = response.json() return VocalState( emotion=data.get("emotion", "neutral"), confidence=data.get("emotion_confidence", 0.0), pitch_mean_hz=data.get("pitch_mean_hz"), energy_rms=data.get("energy_rms", 0.0), speaking_rate_approx=data.get("speaking_rate_approx", 0.0), voiced_fraction=data.get("voiced_fraction", 0.0), ) except Exception as e: logger.warning(f"SpeechBrain analysis failed (continuing without): {e}") return None async def run_voice_coach_pipeline( livekit_url: str, livekit_token: str, anthropic_key: str, deepgram_key: str, elevenlabs_key: str, elevenlabs_voice_id: str, session_context: dict, ): """ Assemble and run the full voice coaching pipeline. session_context contains: baseline_context, session_history, current_mode """ # Transport: LiveKit WebRTC transport = LiveKitTransport( url=livekit_url, token=livekit_token, room_name="voicecoach", params=LiveKitParams( audio_in_enabled=True, audio_out_enabled=True, vad_analyzer=SileroVADAnalyzer(), ), ) # STT: Deepgram streaming with aggressive endpointing for noisy car stt = DeepgramSTTService( api_key=deepgram_key, live_options=LiveOptions( model="nova-2", language="en-US", endpointing=500, # 500ms silence = end of utterance interim_results=False, smart_format=True, punctuate=True, ), ) # Vocal state injection (SpeechBrain) vocal_injector = VocalStateInjector() # LLM: Claude llm = AnthropicLLMService( api_key=anthropic_key, model=ANTHROPIC_MODEL, max_tokens=512, ) # System prompt with context system_prompt = build_system_prompt( baseline_context=session_context.get("baseline_context", ""), session_history=session_context.get("session_history", ""), current_mode=session_context.get("current_mode", "coaching mode"), ) context = OpenAILLMContext( messages=[{"role": "system", "content": system_prompt}] ) context_aggregator = llm.create_context_aggregator(context) # TTS: ElevenLabs streaming tts = ElevenLabsTTSService( api_key=elevenlabs_key, voice_id=elevenlabs_voice_id, model="eleven_turbo_v2_5", # Lowest latency model optimize_streaming_latency=4, ) # Assemble pipeline pipeline = Pipeline([ transport.input(), stt, vocal_injector, context_aggregator.user(), llm, tts, transport.output(), context_aggregator.assistant(), ]) task = PipelineTask( pipeline, params=PipelineParams(allow_interruptions=True), ) runner = PipelineRunner() @transport.event_handler("on_first_participant_joined") async def on_first_participant_joined(transport, participant): logger.info(f"Participant joined: {participant['identity']}") await task.queue_frames([ TextFrame("VoiceCoach connected. How are you doing today?") ]) @transport.event_handler("on_participant_left") async def on_participant_left(transport, participant, reason): logger.info(f"Participant left: {participant['identity']}") await task.queue_frames([EndFrame()]) await runner.run(task) ``` ### `pipecat-agent/main.py` ```python """ Entry point for the Pipecat VoiceCoach agent. Reads secrets from Docker Swarm secret files. """ import asyncio import logging import os from pathlib import Path import structlog from livekit import api as livekit_api from pipeline import run_voice_coach_pipeline from openclaw_client import OpenClawClient logging.basicConfig(level=logging.INFO) logger = structlog.get_logger() def read_secret(name: str, env_fallback: str = None) -> str: """Read from Docker Swarm secret file, fall back to env var.""" secret_path = Path(f"/run/secrets/{name}") if secret_path.exists(): return secret_path.read_text().strip() if env_fallback and os.getenv(env_fallback): return os.getenv(env_fallback) raise RuntimeError(f"Secret '{name}' not found at {secret_path} and no env fallback") async def main(): # Load secrets anthropic_key = read_secret("voicecoach_anthropic_key", "ANTHROPIC_API_KEY") deepgram_key = read_secret("voicecoach_deepgram_key", "DEEPGRAM_API_KEY") elevenlabs_key = read_secret("voicecoach_elevenlabs_key", "ELEVENLABS_API_KEY") elevenlabs_voice_id = read_secret("voicecoach_elevenlabs_voice_id", "ELEVENLABS_VOICE_ID") livekit_api_key = read_secret("voicecoach_livekit_api_key", "LIVEKIT_API_KEY") livekit_api_secret = read_secret("voicecoach_livekit_api_secret", "LIVEKIT_API_SECRET") openclaw_token = read_secret("voicecoach_openclaw_token", "OPENCLAW_TOKEN") livekit_url = os.getenv("LIVEKIT_URL", "ws://livekit-server:7880") openclaw_url = os.getenv("OPENCLAW_URL", "http://openclaw-voicecoach:3000") # Fetch session context from OpenClaw openclaw = OpenClawClient(openclaw_url, openclaw_token) session_context = await openclaw.get_session_context() logger.info("Session context loaded", mode=session_context.get("current_mode")) # Generate LiveKit access token for this agent token = livekit_api.AccessToken(livekit_api_key, livekit_api_secret) token.with_grants(livekit_api.VideoGrants(room_join=True, room="voicecoach")) token.identity = "voicecoach-agent" livekit_token = token.to_jwt() logger.info("Starting VoiceCoach pipeline", livekit_url=livekit_url) await run_voice_coach_pipeline( livekit_url=livekit_url, livekit_token=livekit_token, anthropic_key=anthropic_key, deepgram_key=deepgram_key, elevenlabs_key=elevenlabs_key, elevenlabs_voice_id=elevenlabs_voice_id, session_context=session_context, ) if __name__ == "__main__": asyncio.run(main()) ``` ### `pipecat-agent/openclaw_client.py` ```python """ Client for reading/writing session data to the OpenClaw VoiceCoach agent. """ import httpx import logging logger = logging.getLogger(__name__) class OpenClawClient: def __init__(self, base_url: str, token: str): self.base_url = base_url.rstrip("/") self.headers = {"Authorization": f"Bearer {token}"} async def get_session_context(self) -> dict: """Fetch baseline and recent session history from OpenClaw memory.""" try: async with httpx.AsyncClient(timeout=5.0) as client: r = await client.get( f"{self.base_url}/api/memory", headers=self.headers, params={"keys": "vocal_baseline,recent_sessions,current_mode"}, ) r.raise_for_status() memory = r.json() return { "baseline_context": memory.get("vocal_baseline", ""), "session_history": memory.get("recent_sessions", ""), "current_mode": memory.get("current_mode", "coaching mode"), } except Exception as e: logger.warning(f"Could not fetch OpenClaw context: {e}. Using defaults.") return {"baseline_context": "", "session_history": "", "current_mode": "coaching mode"} async def write_session_summary(self, summary: dict) -> bool: """Write end-of-session summary to OpenClaw memory.""" try: async with httpx.AsyncClient(timeout=5.0) as client: r = await client.post( f"{self.base_url}/api/memory", headers=self.headers, json={"key": "recent_sessions", "value": summary}, ) r.raise_for_status() return True except Exception as e: logger.error(f"Failed to write session summary: {e}") return False ``` ### `pipecat-agent/Dockerfile` ```dockerfile FROM python:3.11-slim RUN apt-get update && apt-get install -y \ libsndfile1 \ ffmpeg \ && rm -rf /var/lib/apt/lists/* WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY . . HEALTHCHECK --interval=30s --timeout=10s --start-period=30s \ CMD python -c "import asyncio; print('pipeline process alive')" || exit 1 CMD ["python", "main.py"] ``` ### `pipecat-agent/tests/test_prompt_builder.py` ```python """Tests for prompt assembly.""" from prompt_builder import build_user_message, VocalState, build_system_prompt def test_vocal_state_injected_into_message(): state = VocalState( emotion="ang", confidence=0.82, pitch_mean_hz=215.0, energy_rms=0.71, speaking_rate_approx=4.8, voiced_fraction=0.91, ) result = build_user_message("I can't get my team to listen", state) assert "[VOCAL_STATE" in result assert "emotion=ang" in result assert "I can't get my team to listen" in result def test_low_confidence_vocal_state_omitted(): state = VocalState( emotion="ang", confidence=0.15, # Below threshold pitch_mean_hz=None, energy_rms=0.0, speaking_rate_approx=0.0, voiced_fraction=0.0, ) result = build_user_message("Hello", state) assert "[VOCAL_STATE" not in result def test_none_vocal_state_handled(): result = build_user_message("Hello", None) assert result == "Hello" def test_system_prompt_contains_mode(): prompt = build_system_prompt(current_mode="vent mode") assert "vent mode" in prompt def test_system_prompt_contains_baseline_when_provided(): prompt = build_system_prompt(baseline_context="Jeremy's baseline pitch: 180hz") assert "180hz" in prompt ``` --- ## Step 4: LiveKit Server Config ### `livekit/livekit.yaml` ```yaml port: 7880 rtc: tcp_port: 7881 port_range_start: 50000 port_range_end: 60000 use_external_ip: true # Required for remote clients (phone from road) # Logging logging: json: false level: info # Room settings room: max_participants: 5 empty_timeout: 300 # Close room after 5min of no participants departure_timeout: 30 # Keys are loaded from environment — injected via Docker secrets # Format: key_name: secret_value (loaded from LIVEKIT_KEYS env var) ``` --- ## Step 5: Web Client ### `web-client/index.html` ```html VoiceCoach

🎙 VoiceCoach

Disconnected
Mode: coaching

Press "Start Session" to connect your voice.

``` ### `web-client/app.js` ```javascript const GATEWAY_URL = window.location.origin + "/api"; let room = null; async function getToken() { const response = await fetch(`${GATEWAY_URL}/token`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ identity: "jeremy-" + Date.now() }), }); if (!response.ok) throw new Error("Failed to get token"); return response.json(); } async function connect() { setStatus("connecting", "Connecting..."); try { const { token, url } = await getToken(); room = new LivekitClient.Room({ audioCaptureDefaults: { echoCancellation: true, noiseSuppression: true, autoGainControl: true, }, }); room.on(LivekitClient.RoomEvent.Connected, () => { setStatus("connected", "Connected"); document.getElementById("connectBtn").style.display = "none"; document.getElementById("disconnectBtn").style.display = "block"; }); room.on(LivekitClient.RoomEvent.Disconnected, () => { setStatus("disconnected", "Disconnected"); document.getElementById("connectBtn").style.display = "block"; document.getElementById("disconnectBtn").style.display = "none"; }); room.on(LivekitClient.RoomEvent.TrackSubscribed, (track) => { if (track.kind === LivekitClient.Track.Kind.Audio) { const element = track.attach(); document.body.appendChild(element); element.play(); } }); await room.connect(url, token); await room.localParticipant.setMicrophoneEnabled(true); } catch (err) { setStatus("disconnected", "Error: " + err.message); console.error(err); } } async function disconnect() { if (room) { await room.disconnect(); room = null; } } function setMode(mode) { document.getElementById("modeDisplay").textContent = "Mode: " + mode; if (room && room.localParticipant) { // Send mode as data message to agent const encoder = new TextEncoder(); room.localParticipant.publishData( encoder.encode(JSON.stringify({ type: "mode_change", mode })), { reliable: true } ); } } function setStatus(state, text) { const el = document.getElementById("status"); el.className = "status " + state; el.textContent = text; } document.getElementById("connectBtn").addEventListener("click", connect); document.getElementById("disconnectBtn").addEventListener("click", disconnect); ``` ### `web-client/Dockerfile` ```dockerfile FROM nginx:alpine COPY index.html /usr/share/nginx/html/ COPY app.js /usr/share/nginx/html/ COPY style.css /usr/share/nginx/html/ # Token endpoint proxy config COPY nginx.conf /etc/nginx/conf.d/default.conf EXPOSE 80 ``` ### `web-client/nginx.conf` ```nginx server { listen 80; root /usr/share/nginx/html; index index.html; # Proxy /api/* to the pipecat agent's token service location /api/ { proxy_pass http://pipecat-agent:8080/; proxy_http_version 1.1; proxy_set_header Host $host; } location / { try_files $uri $uri/ /index.html; } } ``` --- ## Step 6: OpenClaw Agent Config ### `openclaw-agent/openclaw.json` ```json { "name": "VoiceCoach", "description": "Real-time voice coaching agent for commute sessions. Integrates vocal analysis with communication coaching.", "version": "1.0.0", "model": { "provider": "anthropic", "name": "claude-sonnet-4-6", "temperature": 0.7, "max_tokens": 512 }, "memory": { "enabled": true, "keys": [ "vocal_baseline", "recent_sessions", "current_mode", "action_items", "coaching_themes" ] }, "communication": { "text": { "enabled": true }, "voice": { "enabled": false } }, "plugins": { "entries": {} }, "api": { "enabled": true, "port": 3000, "cors": ["http://pipecat-agent:8080"] } } ``` --- ## Step 7: Docker Stack Definition ### `docker-stack.yml` ```yaml version: "3.8" networks: hobbs-overlay: external: true voicecoach-internal: driver: overlay internal: true # Not routable externally volumes: speechbrain-models: driver: local openclaw-voicecoach-data: driver: local livekit-data: driver: local secrets: voicecoach_anthropic_key: external: true voicecoach_deepgram_key: external: true voicecoach_elevenlabs_key: external: true voicecoach_elevenlabs_voice_id: external: true voicecoach_openclaw_token: external: true voicecoach_livekit_api_key: external: true voicecoach_livekit_api_secret: external: true services: # ────────────────────────────────────────────────────────── # LiveKit WebRTC Server # ────────────────────────────────────────────────────────── livekit-server: image: livekit/livekit-server:latest command: --config /etc/livekit/livekit.yaml ports: - "7880:7880" # HTTP/WS API - "7881:7881" # RTC over TCP - "50000-60000:50000-60000/udp" # WebRTC media volumes: - ./livekit/livekit.yaml:/etc/livekit/livekit.yaml:ro - livekit-data:/data environment: - LIVEKIT_KEYS_FILE=/run/secrets/voicecoach_livekit_api_key secrets: - voicecoach_livekit_api_key - voicecoach_livekit_api_secret networks: - hobbs-overlay - voicecoach-internal deploy: replicas: 1 placement: constraints: - node.role == manager # LiveKit benefits from stable placement labels: - "traefik.enable=true" - "traefik.http.routers.livekit.rule=Host(`voice.hobbs.farm`)" - "traefik.http.routers.livekit.entrypoints=websecure" - "traefik.http.routers.livekit.tls.certresolver=cloudflare" - "traefik.http.services.livekit.loadbalancer.server.port=7880" - "traefik.http.routers.livekit.middlewares=livekit-websocket" - "traefik.http.middlewares.livekit-websocket.headers.customrequestheaders.Upgrade=websocket" restart_policy: condition: on-failure # ────────────────────────────────────────────────────────── # SpeechBrain Emotion Analysis Microservice (INTERNAL ONLY) # ────────────────────────────────────────────────────────── speechbrain-service: image: hobbs.farm/voicecoach/speechbrain:latest build: context: ./speechbrain-service volumes: - speechbrain-models:/app/models environment: - MODEL_DIR=/app/models networks: - voicecoach-internal deploy: replicas: 1 placement: constraints: - node.labels.gpu != true # Keep off GPU node unless configured resources: limits: cpus: "2" memory: "3G" reservations: memory: "2G" restart_policy: condition: on-failure healthcheck: test: ["CMD", "python", "-c", "import httpx; httpx.get('http://localhost:8001/health').raise_for_status()"] interval: 30s timeout: 10s retries: 3 start_period: 90s # Model loading takes time # ────────────────────────────────────────────────────────── # Pipecat Voice Pipeline Agent # ────────────────────────────────────────────────────────── pipecat-agent: image: hobbs.farm/voicecoach/pipecat:latest build: context: ./pipecat-agent environment: - LIVEKIT_URL=ws://livekit-server:7880 - SPEECHBRAIN_SERVICE_URL=http://speechbrain-service:8001 - OPENCLAW_URL=http://openclaw-voicecoach:3000 secrets: - voicecoach_anthropic_key - voicecoach_deepgram_key - voicecoach_elevenlabs_key - voicecoach_elevenlabs_voice_id - voicecoach_livekit_api_key - voicecoach_livekit_api_secret - voicecoach_openclaw_token networks: - voicecoach-internal - hobbs-overlay # Needs internet for Anthropic/Deepgram/ElevenLabs APIs depends_on: - livekit-server - speechbrain-service - openclaw-voicecoach deploy: replicas: 1 restart_policy: condition: on-failure delay: 10s # ────────────────────────────────────────────────────────── # OpenClaw VoiceCoach Agent # ────────────────────────────────────────────────────────── openclaw-voicecoach: image: ghcr.io/openclaw/openclaw:latest volumes: - ./openclaw-agent/openclaw.json:/app/openclaw.json:ro - ./openclaw-agent/system_prompt.md:/app/system_prompt.md:ro - ./openclaw-agent/skills:/app/skills:ro - openclaw-voicecoach-data:/app/data secrets: - voicecoach_anthropic_key - voicecoach_openclaw_token environment: - OPENCLAW_CONFIG=/app/openclaw.json - ANTHROPIC_API_KEY_FILE=/run/secrets/voicecoach_anthropic_key networks: - voicecoach-internal - hobbs-overlay deploy: replicas: 1 restart_policy: condition: on-failure healthcheck: test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"] interval: 30s timeout: 5s retries: 3 # ────────────────────────────────────────────────────────── # Web Client (phone interface) # ────────────────────────────────────────────────────────── voicecoach-web: image: hobbs.farm/voicecoach/web:latest build: context: ./web-client networks: - hobbs-overlay - voicecoach-internal deploy: replicas: 1 labels: - "traefik.enable=true" - "traefik.http.routers.voicecoach.rule=Host(`coach.hobbs.farm`)" - "traefik.http.routers.voicecoach.entrypoints=websecure" - "traefik.http.routers.voicecoach.tls.certresolver=cloudflare" - "traefik.http.services.voicecoach.loadbalancer.server.port=80" - "traefik.http.routers.voicecoach.middlewares=voicecoach-auth" - "traefik.http.middlewares.voicecoach-auth.basicauth.users=jeremy:$$apr1$$REPLACEME" restart_policy: condition: on-failure ``` > **Note for Claude Code:** Replace `$$apr1$$REPLACEME` in the basicauth label with the output of: > ```bash > htpasswd -nb jeremy yourpassword | sed 's/\$/\$\$/g' > ``` --- ## Step 8: Makefile ### `Makefile` ```makefile .PHONY: build deploy secrets test smoke e2e clean logs # Build all images build: docker build -t hobbs.farm/voicecoach/speechbrain:latest ./speechbrain-service docker build -t hobbs.farm/voicecoach/pipecat:latest ./pipecat-agent docker build -t hobbs.farm/voicecoach/web:latest ./web-client # Create secrets (run once) secrets: bash scripts/create-secrets.sh # Deploy stack deploy: build docker stack deploy -c docker-stack.yml voicecoach # Run unit tests (local, no Docker required) test: cd speechbrain-service && python -m pytest tests/ -v cd pipecat-agent && python -m pytest tests/ -v # Quick smoke test after deploy smoke: bash scripts/smoke-test.sh # Full end-to-end test e2e: bash scripts/e2e-test.sh # Follow all service logs logs: docker service logs voicecoach_pipecat-agent -f # Remove stack clean: docker stack rm voicecoach ``` --- ## Step 9: Test & Validation Scripts ### `scripts/smoke-test.sh` ```bash #!/bin/bash # Quick validation — run immediately after deploy set -e echo "=== VoiceCoach Smoke Tests ===" # 1. Check all services running echo "" echo "→ Checking service replicas..." docker service ls | grep voicecoach FAILED=$(docker service ls | grep voicecoach | grep "0/" | wc -l) if [ "$FAILED" -gt 0 ]; then echo "❌ Some services have 0 replicas. Check: docker service ps voicecoach_" exit 1 fi echo "✅ All services running" # 2. LiveKit health echo "" echo "→ LiveKit health check..." curl -sf https://voice.hobbs.farm/rtc/health > /dev/null && echo "✅ LiveKit responding" || echo "❌ LiveKit not responding" # 3. SpeechBrain health (via internal overlay — run from swarm manager) echo "" echo "→ SpeechBrain health check (internal)..." # Run from a container on the voicecoach-internal network docker run --rm --network voicecoach_voicecoach-internal \ curlimages/curl:latest \ curl -sf http://speechbrain-service:8001/health && echo "✅ SpeechBrain responding" || echo "❌ SpeechBrain not responding" # 4. Web client echo "" echo "→ Web client check..." curl -sf -u jeremy:yourpassword https://coach.hobbs.farm > /dev/null && echo "✅ Web client responding" || echo "❌ Web client not responding" # 5. OpenClaw health echo "" echo "→ OpenClaw health check (internal)..." docker run --rm --network voicecoach_voicecoach-internal \ curlimages/curl:latest \ curl -sf http://openclaw-voicecoach:3000/health && echo "✅ OpenClaw responding" || echo "❌ OpenClaw not responding" echo "" echo "=== Smoke tests complete ===" ``` ### `scripts/e2e-test.sh` ```bash #!/bin/bash # Full end-to-end test using a pre-recorded test WAV # Requires: livekit-cli installed (go install github.com/livekit/livekit-cli/cmd/lk@latest) echo "=== VoiceCoach End-to-End Test ===" echo "This test joins the LiveKit room as a test participant and" echo "sends a pre-recorded audio clip, then checks for TTS output." LK_KEY=$(docker secret inspect voicecoach_livekit_api_key --format '{{.Spec.Name}}' 2>/dev/null || echo "not-found") echo "" echo "→ Downloading test audio..." curl -sL "https://www2.cs.uic.edu/~i101/SoundFiles/StarWars60.wav" -o /tmp/test_audio.wav echo "" echo "→ Joining LiveKit room and sending test audio..." echo " (Manual step: open https://coach.hobbs.farm on phone and verify you hear a response)" echo "" echo "→ Alternatively, use livekit-cli:" echo " lk room join --url wss://voice.hobbs.farm --api-key \$KEY --api-secret \$SECRET voicecoach" echo "" echo "=== E2E test requires manual verification at this time ===" echo "→ Automated e2e test with audio injection coming in v1.1" ``` ### `tests/test_latency.sh` ```bash #!/bin/bash # Measure SpeechBrain analysis latency echo "=== Latency Test: SpeechBrain Analysis ===" # Download a 2-second speech sample curl -sL "https://www2.cs.uic.edu/~i101/SoundFiles/preamble.wav" -o /tmp/latency_test.wav # Test via internal network START=$(date +%s%3N) docker run --rm --network voicecoach_voicecoach-internal \ -v /tmp/latency_test.wav:/test.wav \ curlimages/curl:latest \ curl -sf -X POST http://speechbrain-service:8001/analyze \ -F "audio=@/test.wav" > /tmp/latency_result.json END=$(date +%s%3N) ELAPSED=$((END - START)) echo "Analysis latency: ${ELAPSED}ms" if [ "$ELAPSED" -lt 3000 ]; then echo "✅ Within 3s budget" else echo "⚠️ Exceeds 3s budget — consider GPU or lighter model" fi echo "" cat /tmp/latency_result.json | python3 -m json.tool ``` --- ## Step 10: Deploy Order Claude Code should execute in this exact order: ```bash # 1. Clone/create repository structure # (Claude Code: create all files from this document) # 2. Create Docker Swarm secrets (ONCE) make secrets # 3. Confirm overlay network docker network ls | grep hobbs-overlay || docker network create --driver overlay --attachable hobbs-overlay # 4. Build images make build # 5. Deploy stack make deploy # 6. Monitor startup (SpeechBrain takes ~90s to load model) watch docker service ls # 7. Run smoke tests make smoke # 8. Run unit tests make test # 9. Open coach.hobbs.farm on phone and test voice ``` --- ## Interface Change Summary **What changes from your current Matrix setup:** | Capability | Before | After | |---|---|---| | Real-time voice with agents | ❌ Not supported | ✅ coach.hobbs.farm (phone browser) | | Text chat with Pluto/agents | ✅ Matrix stays | ✅ Matrix stays (unchanged) | | Async task dispatch | ✅ Matrix/Dispatch | ✅ Matrix/Dispatch stays | | Vocal analysis coaching | ❌ Not possible | ✅ Full pipeline | | Session memory | ✅ OpenClaw memory | ✅ OpenClaw memory (enhanced) | **Matrix is NOT replaced.** It remains your primary async interface for Pluto and all text-based agent interaction. VoiceCoach is additive — a dedicated real-time voice interface for commute sessions only. --- ## Rollback ```bash # Remove the voice coach stack entirely (OpenClaw memory data preserved in volume) docker stack rm voicecoach # Remove secrets if rotating docker secret rm voicecoach_anthropic_key voicecoach_deepgram_key \ voicecoach_elevenlabs_key voicecoach_elevenlabs_voice_id \ voicecoach_openclaw_token voicecoach_livekit_api_key voicecoach_livekit_api_secret ``` --- ## Known Issues & Troubleshooting **SpeechBrain takes too long on CPU:** - First inference is always slower (model warmup). Subsequent calls are faster. - If consistently > 3s: add a lighter model path using librosa-only analysis (no SpeechBrain) as fallback. **WebRTC media not flowing (hear nothing):** - Almost certainly firewall/NAT. Confirm UDP 50000-60000 is open at router. - LiveKit logs: `docker service logs voicecoach_livekit-server -f` **Deepgram returning no transcripts:** - Check API key secret. Deepgram errors appear in pipecat-agent logs. - Try `endpointing=1000` if utterances are being cut off. **ElevenLabs high latency:** - Switch to `eleven_turbo_v2_5` model (already set). If still slow, use `edge-tts` locally. - Set `optimize_streaming_latency=4` (already set — maximum optimization). **SpeechBrain emotion always "neutral":** - Normal for the IEMOCAP model on non-actor speech. Acoustic features (pitch, energy, rate) are more reliable than the emotion label for real speech. Rely on those for coaching signals.