hobbs-iac/VOICECOACH_DEPLOY.md

VoiceCoach — Homelab Deployment Instructions

Pipecat + SpeechBrain + Claude + LiveKit + OpenClaw

Target: hobbs.farm Docker Swarm | Author: Claude | For use with Claude Code

---

Requirements Iteration Log

Claude Code: Do NOT skip this section. It documents why decisions were made and will save you from dead ends.

Iteration 1 — Core Functional Requirements (Raw)

Initial capture of what this system must do:

• Receive real-time voice from Jeremy's phone while driving
• Transcribe speech to text
• Analyze vocal characteristics (pitch, energy, emotion, speaking rate)
• Pass both transcript AND vocal metadata to Claude
• Claude responds with coaching feedback delivered as spoken audio
• Session memory persists across commutes
• Self-hosted on hobbs.farm infrastructure

Gap identified: No interface specified. Matrix is async/text-focused — not suitable for real-time bidirectional audio streaming. Need a WebRTC solution.

---

Iteration 2 — Infrastructure Constraint Pass

Applying Jeremy's known infrastructure to the requirements:

• Must deploy as Docker Swarm stack — nodes: hf-pdocker-01, hf-pdocker-02, bart
• Must use Traefik for SSL termination with existing wildcard *.hobbs.farm cert via Cloudflare DNS challenge
• Must integrate with OpenClaw — new agent instance, not a standalone app
• Must store conversation memory in OpenClaw's memory system (compatible with Pluto)
• Interface decision: Replace Matrix voice with LiveKit (self-hosted WebRTC) at voice.hobbs.farm, web client at coach.hobbs.farm. Matrix retained for text/async agent interaction only.
• SpeechBrain must run as its own microservice — not embedded in pipeline — so it can be reused and scaled independently
• No GPU assumption for initial deploy — SpeechBrain runs on CPU. GPU path documented but optional.

Gap identified: Need to define the exact data contract between SpeechBrain service and Pipecat pipeline. What fields get passed to Claude and in what format.

---

Iteration 3 — SpeechBrain & Emotion Pipeline Detail

Defining the data contract and model choices:

SpeechBrain Model: speechbrain/emotion-recognition-wav2vec2-IEMOCAP

• Classifies: neutral, happy, sad, angry, frustrated
• Input: 16kHz mono WAV, minimum ~1 second
• Output: emotion label + per-class confidence scores

Additional acoustic features via librosa:

• pitch_mean_hz — fundamental frequency average
• pitch_variance — pitch stability (high variance = animated/stressed)
• energy_rms — loudness/intensity
• speaking_rate_syllables_per_sec — estimated via onset detection
• voiced_fraction — ratio of voiced to unvoiced frames (low = hesitant/uncertain)

Prompt injection format (into every Claude turn):

[VOCAL_STATE emotion=frustrated confidence=0.82 pitch=215hz pitch_var=high energy=0.71 rate=4.8sps voiced=0.91]
User said: "I just cannot get the team to listen to me in standups"

Claude system prompt must instruct:

• Interpret [VOCAL_STATE] tags as paralinguistic context
• Distinguish between what the user says and how they sound saying it
• Provide coaching on both content and delivery when relevant
• Never mention the raw numbers — translate to natural coaching language

Gap identified: How does Pipecat know when an utterance is complete to trigger SpeechBrain? Need end-of-turn detection strategy. Deepgram handles this via is_final events in streaming mode.

---

Iteration 4 — Testing, Validation, Security

Defining what "working" means before writing code:

Unit Tests:

• test_speechbrain_service.py — POST audio file, assert emotion field present, confidence > 0, latency < 2s
• test_pipecat_pipeline.py — mock Deepgram + SpeechBrain, assert Claude receives correctly formatted prompt
• test_openclaw_agent.py — assert agent responds and memory is written after session

Integration Tests:

• test_e2e_pipeline.sh — send WAV file to LiveKit room via livekit-cli, assert TTS audio received back
• test_latency.sh — measure total roundtrip from audio-in to audio-out, assert < 4s

Docker Health Checks:

• SpeechBrain service: GET /health returns 200, model loaded
• Pipecat agent: process alive check
• LiveKit: GET /rtc/health

Validation Steps (run before marking deploy complete):

1. docker service ls — all 6 services showing replicas as desired
2. curl https://voice.hobbs.farm/rtc/health — LiveKit responding
3. curl https://coach.hobbs.farm — web client loads
4. SpeechBrain smoke test with provided sample audio
5. Full end-to-end: open coach.hobbs.farm on phone, say "test", hear Claude respond

Security:

• LiveKit API key/secret stored as Docker Swarm secrets (not env vars)
• Deepgram API key as Swarm secret
• Anthropic API key as Swarm secret
• ElevenLabs API key as Swarm secret
• coach.hobbs.farm protected by basic auth via Traefik middleware (single user)
• SpeechBrain service NOT exposed externally — internal Swarm overlay network only
• OpenClaw agent token as Swarm secret

Gap identified: Need to address baseline calibration — SpeechBrain trained on IEMOCAP actors, not Jeremy's voice. First session should establish Jeremy's neutral baseline so relative changes are meaningful.

---

Iteration 5 — Memory, Continuity & Calibration

Addressing the coaching quality requirements:

Baseline Calibration (first session only):

• First 60 seconds of first session: Claude prompts Jeremy to read a neutral calibration script
• SpeechBrain readings stored in OpenClaw memory as vocal_baseline profile
• Subsequent sessions: all readings compared to baseline, delta reported to Claude
• Format: [VOCAL_STATE emotion=frustrated delta_from_baseline: pitch+18hz energy+0.12]

Session Memory Structure in OpenClaw:

{
  "session_id": "2026-04-01-commute-am",
  "duration_minutes": 22,
  "topics_discussed": ["team communication", "standup facilitation"],
  "vocal_trends": {
    "avg_stress_level": 0.71,
    "peak_frustration_at": "12:34",
    "improvement_noted": false
  },
  "coaching_notes": "Jeremy shows elevated pitch when discussing team dynamics",
  "action_items": ["Practice lower speaking rate in next standup"]
}

Cross-session coaching continuity:

• OpenClaw system prompt includes last 3 session summaries
• Claude tracks whether previous action items were attempted
• Vocal improvement over time is explicitly tracked and acknowledged

Commute-optimized coaching modes (user-selectable via wake phrase):

• "coaching mode" — deep conversation, feedback on every utterance
• "vent mode" — Claude listens, reflects back, minimal interruption
• "practice mode" — Claude plays a difficult stakeholder, Jeremy practices
• "briefing mode" — Claude summarizes pending tasks from Pluto/OpenClaw memory

Gap identified: Need to define how mode switching works mid-conversation without typing. Answer: wake phrases detected at STT layer, routed before Claude prompt.

---

Iteration 6 — Latency Budget, Fallbacks & Observability

Ensuring the system is actually usable in a moving truck:

Latency Budget (target < 3.5s total roundtrip):

• Deepgram streaming STT end-of-turn detection: ~200ms
• SpeechBrain analysis (async, overlaps with STT): ~800ms on CPU
• Claude API call: ~1200ms
• ElevenLabs TTS first audio chunk: ~400ms
• LiveKit transmission: ~100ms
• Total: ~2.7s — within target

Fallbacks:

• If SpeechBrain service unreachable: pipeline continues without vocal metadata, logs warning
• If Deepgram fails: fallback to Whisper local STT (slower but self-contained)
• If ElevenLabs fails: fallback to edge-tts (local, lower quality)
• If Claude API fails: return "I'm having trouble connecting, please try again"

Observability:

• All services log to Docker's logging driver
• Pipecat pipeline emits structured JSON logs per utterance including latencies
• SpeechBrain service logs emotion distribution per session (useful for trend analysis)
• Consider forwarding to existing monitoring stack if Prometheus/Grafana present

Mobile network resilience:

• LiveKit handles packet loss via WebRTC NACK/FEC automatically
• Configure Deepgram with endpointing=500 (500ms silence = end of utterance) — more forgiving for noisy truck cab
• Push-to-talk fallback mode available via coach.hobbs.farm UI (hold button while speaking)

---

Iteration 7 — Claude Code Readiness Pass

Final pass ensuring Claude Code can execute this without ambiguity:

Explicit file structure defined — Claude Code should create exactly these files All environment variables listed with descriptions and example values Each service has its own section with Dockerfile, config, and test Deploy order specified — secrets → network → redis → speechbrain → livekit → pipecat → openclaw → web client Rollback procedure documented Every docker stack deploy command is copy-pasteable Tests are runnable with a single command after deploy

---

System Architecture

┌─────────────────────────────────────────────────────────┐
│                    PHONE (in truck)                      │
│         coach.hobbs.farm (Chrome/Safari browser)         │
│         LiveKit JS SDK — WebRTC audio stream             │
└──────────────────────┬──────────────────────────────────┘
                       │ WebRTC (DTLS/SRTP)
                       ▼
┌─────────────────────────────────────────────────────────┐
│              LiveKit Server (voice.hobbs.farm)            │
│              WebRTC SFU — routes audio frames            │
└──────────┬───────────────────────────────────┬──────────┘
           │ LiveKit SDK (internal)             │ LiveKit SDK
           ▼                                   ▼
┌──────────────────────┐             ┌──────────────────────┐
│   Pipecat Pipeline   │             │  (future: recording) │
│   Agent Process      │             └──────────────────────┘
│                      │
│  ┌────────────────┐  │
│  │ Deepgram STT   │  │  ← streaming audio chunks
│  │ (streaming)    │  │
│  └───────┬────────┘  │
│          │ transcript │
│          ▼            │
│  ┌────────────────┐  │
│  │ SpeechBrain    │──┼──→ POST /analyze (internal overlay network)
│  │ Emotion Client │  │  ← {emotion, pitch, energy, rate, voiced}
│  └───────┬────────┘  │
│          │            │
│          ▼            │
│  ┌────────────────┐  │
│  │ Claude API     │  │  ← formatted prompt with [VOCAL_STATE]
│  │ claude-sonnet  │  │  → coaching response text
│  └───────┬────────┘  │
│          │            │
│          ▼            │
│  ┌────────────────┐  │
│  │ ElevenLabs TTS │  │
│  │ (streaming)    │  │
│  └───────┬────────┘  │
│          │ audio      │
└──────────┼────────────┘
           │ audio frames back to LiveKit
           ▼
      Phone speaker
           
           
┌──────────────────────────────────────────────┐
│          SpeechBrain Microservice             │
│          (internal network only)              │
│  FastAPI + wav2vec2-IEMOCAP + librosa         │
│  POST /analyze → emotion + prosody JSON       │
│  GET  /health  → 200 OK                       │
│  GET  /calibrate → session baseline           │
└──────────────────────────────────────────────┘

┌──────────────────────────────────────────────┐
│          OpenClaw VoiceCoach Agent            │
│  - Manages session lifecycle                  │
│  - Writes memory after each session           │
│  - Exposes /session/start, /session/end       │
│  - Integrates with Pluto's memory store       │
└──────────────────────────────────────────────┘

---

Repository Structure

Claude Code should create this exact directory structure:

voicecoach/
├── docker-stack.yml                    # Main Swarm stack definition
├── .env.example                        # Template for secrets (never commit .env)
├── Makefile                            # Convenience targets
│
├── speechbrain-service/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── main.py                         # FastAPI app
│   ├── analyzer.py                     # SpeechBrain + librosa logic
│   ├── models/                         # Model cache (mount as volume)
│   └── tests/
│       ├── test_analyzer.py
│       └── fixtures/
│           ├── neutral_sample.wav
│           ├── angry_sample.wav
│           └── stressed_sample.wav
│
├── pipecat-agent/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── main.py                         # Pipecat pipeline entry
│   ├── pipeline.py                     # Pipeline assembly
│   ├── prompt_builder.py               # Builds Claude prompts with vocal state
│   ├── mode_detector.py                # Wake phrase detection
│   ├── openclaw_client.py              # Writes session memory to OpenClaw
│   └── tests/
│       ├── test_pipeline.py
│       └── test_prompt_builder.py
│
├── openclaw-agent/
│   ├── openclaw.json                   # OpenClaw agent config
│   ├── system_prompt.md                # VoiceCoach persona & instructions
│   └── skills/
│       └── vocal_memory.md             # Skill for reading/writing vocal baselines
│
├── web-client/
│   ├── Dockerfile                      # nginx serving static files
│   ├── index.html
│   ├── app.js                          # LiveKit JS SDK integration
│   └── style.css
│
├── livekit/
│   └── livekit.yaml                    # LiveKit server config
│
├── scripts/
│   ├── create-secrets.sh               # Creates all Docker Swarm secrets
│   ├── smoke-test.sh                   # Quick validation after deploy
│   ├── e2e-test.sh                     # Full end-to-end test
│   └── calibrate.sh                    # Trigger baseline calibration session
│
└── tests/
    └── test_latency.sh                 # Roundtrip latency measurement

---

Step 0: Prerequisites Checklist

Before Claude Code runs any deployment commands, verify:

# 1. Confirm Swarm manager access
docker node ls

# 2. Confirm Traefik is running with wildcard cert
docker service ls | grep traefik

# 3. Confirm overlay network exists (or create it)
docker network ls | grep hobbs-overlay
# If missing:
docker network create --driver overlay --attachable hobbs-overlay

# 4. Confirm required external API keys are available
# - Anthropic API key (claude-sonnet-4-6)
# - Deepgram API key (https://console.deepgram.com)
# - ElevenLabs API key (https://elevenlabs.io)
# - LiveKit will use self-generated key pair (no external account needed)

# 5. Confirm ports available
# - 7880/tcp  LiveKit HTTPS API
# - 7881/tcp  LiveKit RTC over TCP fallback
# - 50000-60000/udp  LiveKit WebRTC media (must be open on firewall/router)

IMPORTANT: If ports 50000-60000 UDP are not open at the router level, WebRTC media will not flow. This is the most common failure point. Confirm with:

# From external network (not homelab):
nmap -sU -p 50000-50010 <your-public-ip>

---

Step 1: Create Docker Swarm Secrets

File: scripts/create-secrets.sh

#!/bin/bash
# Run this ONCE on the Swarm manager node.
# Have your API keys ready before running.
set -e

echo "Creating Docker Swarm secrets for VoiceCoach..."

read -s -p "Anthropic API Key: " ANTHROPIC_KEY && echo
read -s -p "Deepgram API Key: " DEEPGRAM_KEY && echo
read -s -p "ElevenLabs API Key: " ELEVENLABS_KEY && echo
read -s -p "ElevenLabs Voice ID: " ELEVENLABS_VOICE_ID && echo
read -s -p "OpenClaw Agent Token: " OPENCLAW_TOKEN && echo

# Generate LiveKit key pair
LIVEKIT_API_KEY="voicecoach-$(openssl rand -hex 4)"
LIVEKIT_API_SECRET=$(openssl rand -base64 32)

echo "$ANTHROPIC_KEY"     | docker secret create voicecoach_anthropic_key -
echo "$DEEPGRAM_KEY"      | docker secret create voicecoach_deepgram_key -
echo "$ELEVENLABS_KEY"    | docker secret create voicecoach_elevenlabs_key -
echo "$ELEVENLABS_VOICE_ID" | docker secret create voicecoach_elevenlabs_voice_id -
echo "$OPENCLAW_TOKEN"    | docker secret create voicecoach_openclaw_token -
echo "$LIVEKIT_API_KEY"   | docker secret create voicecoach_livekit_api_key -
echo "$LIVEKIT_API_SECRET" | docker secret create voicecoach_livekit_api_secret -

echo ""
echo "✅ Secrets created. LiveKit credentials:"
echo "   API Key:    $LIVEKIT_API_KEY"
echo "   API Secret: $LIVEKIT_API_SECRET"
echo ""
echo "⚠️  Save these LiveKit credentials — they are stored in Swarm secrets"
echo "    and cannot be retrieved again."

---

Step 2: SpeechBrain Microservice

speechbrain-service/requirements.txt

fastapi==0.115.0
uvicorn[standard]==0.30.0
speechbrain==1.0.0
torch==2.3.0
torchaudio==2.3.0
librosa==0.10.2
soundfile==0.12.1
numpy==1.26.4
httpx==0.27.0
python-multipart==0.0.9

speechbrain-service/analyzer.py

"""
SpeechBrain + librosa vocal analysis.
Performs emotion classification and acoustic feature extraction from audio.
"""
import io
import logging
import numpy as np
import soundfile as sf
import librosa
from speechbrain.inference.interfaces import foreign_class
from dataclasses import dataclass, asdict
from typing import Optional

logger = logging.getLogger(__name__)

SAMPLE_RATE = 16000
MIN_DURATION_SECONDS = 0.5  # Reject clips shorter than this


@dataclass
class VocalAnalysis:
    emotion: str
    emotion_confidence: float
    emotion_scores: dict          # All class probabilities
    pitch_mean_hz: Optional[float]
    pitch_variance: Optional[float]
    energy_rms: float
    speaking_rate_approx: float   # Onset events per second (proxy for syllable rate)
    voiced_fraction: float        # 0-1, fraction of voiced frames
    duration_seconds: float
    analysis_error: Optional[str] = None


class VocalAnalyzer:
    """Loads models once at startup and provides analysis per audio chunk."""

    def __init__(self, model_dir: str = "/app/models"):
        logger.info("Loading SpeechBrain emotion model...")
        self.classifier = foreign_class(
            source="speechbrain/emotion-recognition-wav2vec2-IEMOCAP",
            pymodule_file="custom_interface.py",
            classname="CustomEncoderWav2vec2Classifier",
            savedir=model_dir,
            run_opts={"device": "cpu"},  # Change to "cuda" if GPU available
        )
        logger.info("SpeechBrain model loaded successfully.")

    def analyze(self, audio_bytes: bytes, content_type: str = "audio/wav") -> VocalAnalysis:
        """
        Analyze audio bytes and return vocal features.
        
        Args:
            audio_bytes: Raw audio file bytes (WAV, FLAC, or OGG)
            content_type: MIME type hint
            
        Returns:
            VocalAnalysis dataclass
        """
        try:
            # Load audio
            audio_buf = io.BytesIO(audio_bytes)
            audio, sr = sf.read(audio_buf, dtype="float32")

            # Ensure mono
            if audio.ndim > 1:
                audio = audio.mean(axis=1)

            # Resample to 16kHz if needed
            if sr != SAMPLE_RATE:
                audio = librosa.resample(audio, orig_sr=sr, target_sr=SAMPLE_RATE)
                sr = SAMPLE_RATE

            duration = len(audio) / sr

            if duration < MIN_DURATION_SECONDS:
                return VocalAnalysis(
                    emotion="neutral",
                    emotion_confidence=0.0,
                    emotion_scores={},
                    pitch_mean_hz=None,
                    pitch_variance=None,
                    energy_rms=0.0,
                    speaking_rate_approx=0.0,
                    voiced_fraction=0.0,
                    duration_seconds=duration,
                    analysis_error=f"Audio too short: {duration:.2f}s < {MIN_DURATION_SECONDS}s minimum",
                )

            # --- SpeechBrain emotion classification ---
            import tempfile, os
            with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
                sf.write(tmp.name, audio, sr)
                tmp_path = tmp.name

            try:
                out_prob, score, index, text_lab = self.classifier.classify_file(tmp_path)
                emotion = text_lab[0] if isinstance(text_lab, list) else text_lab
                confidence = float(score[0]) if hasattr(score, "__iter__") else float(score)
                # Build score dict from probability tensor
                labels = ["neu", "hap", "sad", "ang", "fru"]  # IEMOCAP classes
                prob_list = out_prob[0].tolist() if hasattr(out_prob[0], "tolist") else list(out_prob[0])
                emotion_scores = {l: round(p, 4) for l, p in zip(labels, prob_list)}
            finally:
                os.unlink(tmp_path)

            # --- Librosa acoustic features ---
            # Pitch via pyin (more accurate than yin for speech)
            f0, voiced_flag, _ = librosa.pyin(
                audio,
                fmin=librosa.note_to_hz("C2"),
                fmax=librosa.note_to_hz("C7"),
                sr=sr,
            )

            voiced_f0 = f0[voiced_flag] if voiced_flag is not None else np.array([])
            pitch_mean = float(np.mean(voiced_f0)) if len(voiced_f0) > 0 else None
            pitch_var = float(np.var(voiced_f0)) if len(voiced_f0) > 1 else None
            voiced_fraction = float(np.sum(voiced_flag) / len(voiced_flag)) if voiced_flag is not None else 0.0

            # Energy
            rms = librosa.feature.rms(y=audio)
            energy_rms = float(np.mean(rms))

            # Speaking rate proxy: onset strength events per second
            onset_frames = librosa.onset.onset_detect(y=audio, sr=sr)
            speaking_rate = len(onset_frames) / duration if duration > 0 else 0.0

            return VocalAnalysis(
                emotion=emotion,
                emotion_confidence=round(confidence, 4),
                emotion_scores=emotion_scores,
                pitch_mean_hz=round(pitch_mean, 2) if pitch_mean else None,
                pitch_variance=round(pitch_var, 2) if pitch_var else None,
                energy_rms=round(energy_rms, 4),
                speaking_rate_approx=round(speaking_rate, 2),
                voiced_fraction=round(voiced_fraction, 4),
                duration_seconds=round(duration, 3),
            )

        except Exception as e:
            logger.exception(f"Analysis failed: {e}")
            return VocalAnalysis(
                emotion="neutral",
                emotion_confidence=0.0,
                emotion_scores={},
                pitch_mean_hz=None,
                pitch_variance=None,
                energy_rms=0.0,
                speaking_rate_approx=0.0,
                voiced_fraction=0.0,
                duration_seconds=0.0,
                analysis_error=str(e),
            )

speechbrain-service/main.py

"""
FastAPI service wrapping VocalAnalyzer.
Internal-only — not exposed via Traefik.
"""
import logging
import os
from contextlib import asynccontextmanager
from fastapi import FastAPI, UploadFile, File, HTTPException
from fastapi.responses import JSONResponse

from analyzer import VocalAnalyzer, asdict

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

MODEL_DIR = os.getenv("MODEL_DIR", "/app/models")

analyzer: VocalAnalyzer = None


@asynccontextmanager
async def lifespan(app: FastAPI):
    global analyzer
    logger.info("Starting VocalAnalyzer...")
    analyzer = VocalAnalyzer(model_dir=MODEL_DIR)
    logger.info("VocalAnalyzer ready.")
    yield
    logger.info("Shutting down.")


app = FastAPI(title="SpeechBrain Vocal Analysis Service", lifespan=lifespan)


@app.get("/health")
async def health():
    if analyzer is None:
        raise HTTPException(status_code=503, detail="Model not loaded")
    return {"status": "ok", "model": "emotion-recognition-wav2vec2-IEMOCAP"}


@app.post("/analyze")
async def analyze(audio: UploadFile = File(...)):
    """
    Analyze vocal characteristics of an audio clip.
    
    Accepts: WAV, FLAC, OGG audio files.
    Returns: Emotion classification + acoustic features.
    """
    if analyzer is None:
        raise HTTPException(status_code=503, detail="Model not loaded")

    content = await audio.read()
    if len(content) == 0:
        raise HTTPException(status_code=400, detail="Empty audio file")

    result = analyzer.analyze(content, content_type=audio.content_type or "audio/wav")
    return JSONResponse(content=asdict(result))


@app.post("/calibrate")
async def calibrate(audio: UploadFile = File(...)):
    """
    Analyze audio and return it labeled as baseline calibration.
    Same as /analyze but semantically marks this as a neutral reference.
    """
    result_response = await analyze(audio)
    data = result_response.body
    import json
    result_dict = json.loads(data)
    result_dict["is_baseline"] = True
    return JSONResponse(content=result_dict)

speechbrain-service/Dockerfile

FROM python:3.11-slim

# System deps for librosa/soundfile
RUN apt-get update && apt-get install -y \
    libsndfile1 \
    ffmpeg \
    git \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Pre-download the SpeechBrain model at build time to avoid cold start
# This bakes the model into the image (~400MB)
RUN python -c "
from speechbrain.inference.interfaces import foreign_class
foreign_class(
    source='speechbrain/emotion-recognition-wav2vec2-IEMOCAP',
    pymodule_file='custom_interface.py',
    classname='CustomEncoderWav2vec2Classifier',
    savedir='/app/models',
    run_opts={'device': 'cpu'},
)
print('Model downloaded successfully.')
"

COPY . .

EXPOSE 8001

HEALTHCHECK --interval=30s --timeout=10s --start-period=60s \
    CMD python -c "import httpx; httpx.get('http://localhost:8001/health').raise_for_status()"

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8001", "--workers", "1"]

speechbrain-service/tests/test_analyzer.py

"""
Tests for the SpeechBrain vocal analyzer.
Requires test WAV fixtures to be present.
"""
import os
import pytest
import numpy as np
import soundfile as sf
from pathlib import Path

FIXTURES = Path(__file__).parent / "fixtures"


def create_sine_wav(path: Path, frequency: float = 440.0, duration: float = 2.0, sr: int = 16000):
    """Generate a sine wave WAV file for testing."""
    t = np.linspace(0, duration, int(sr * duration))
    audio = 0.5 * np.sin(2 * np.pi * frequency * t).astype(np.float32)
    sf.write(str(path), audio, sr)


@pytest.fixture(scope="module", autouse=True)
def create_fixtures():
    FIXTURES.mkdir(exist_ok=True)
    create_sine_wav(FIXTURES / "neutral_sample.wav", frequency=200, duration=2.0)
    create_sine_wav(FIXTURES / "short_sample.wav", frequency=200, duration=0.2)


def test_analyzer_loads():
    """Analyzer must load without error."""
    from analyzer import VocalAnalyzer
    a = VocalAnalyzer()
    assert a is not None


def test_analyze_returns_emotion():
    from analyzer import VocalAnalyzer
    a = VocalAnalyzer()
    audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes()
    result = a.analyze(audio_bytes)
    assert result.emotion in ["neu", "hap", "sad", "ang", "fru"]
    assert 0.0 <= result.emotion_confidence <= 1.0


def test_analyze_returns_acoustic_features():
    from analyzer import VocalAnalyzer
    a = VocalAnalyzer()
    audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes()
    result = a.analyze(audio_bytes)
    assert result.energy_rms >= 0.0
    assert result.duration_seconds > 0.0
    assert 0.0 <= result.voiced_fraction <= 1.0


def test_short_audio_handled_gracefully():
    from analyzer import VocalAnalyzer
    a = VocalAnalyzer()
    audio_bytes = (FIXTURES / "short_sample.wav").read_bytes()
    result = a.analyze(audio_bytes)
    assert result.analysis_error is not None
    assert "too short" in result.analysis_error.lower()


def test_analyze_latency():
    """Analysis should complete in under 3 seconds on CPU."""
    import time
    from analyzer import VocalAnalyzer
    a = VocalAnalyzer()
    audio_bytes = (FIXTURES / "neutral_sample.wav").read_bytes()
    start = time.time()
    a.analyze(audio_bytes)
    elapsed = time.time() - start
    assert elapsed < 3.0, f"Analysis took {elapsed:.2f}s — exceeds 3s budget"

---

Step 3: Pipecat Voice Pipeline Agent

pipecat-agent/requirements.txt

pipecat-ai[deepgram,elevenlabs,livekit,silero]==0.0.48
anthropic==0.34.0
httpx==0.27.0
python-dotenv==1.0.1
structlog==24.4.0

pipecat-agent/prompt_builder.py

"""
Builds Claude prompts with embedded vocal state metadata.
"""
from dataclasses import dataclass
from typing import Optional


COACHING_SYSTEM_PROMPT = """You are a personal voice coach and communication advisor named VoiceCoach.
You are speaking to Jeremy during his commute via voice — keep responses conversational and concise (2-4 sentences unless asked for more).

You receive a [VOCAL_STATE] tag before each user message containing paralinguistic analysis of Jeremy's voice:
- emotion: the detected emotional state
- confidence: how confident the analysis is (0-1)  
- pitch: average fundamental frequency in Hz (higher = more animated/stressed)
- energy: vocal energy level (0-1, higher = louder/more forceful)
- rate: speaking rate in onset events/sec (higher = faster speech)
- voiced: fraction of voiced frames (lower = more hesitant/uncertain)

Use this information to provide coaching on BOTH what Jeremy says AND how he sounds saying it.
Important: Never quote the raw numbers. Translate them into natural observations.
Example: "I notice some tension in your voice..." not "Your pitch was 215hz..."

{baseline_context}

{session_history}

Available coaching modes (Jeremy can switch by saying the phrase):
- "coaching mode" (default) — active feedback on content and delivery
- "vent mode" — listen and reflect, minimal advice
- "practice mode" — roleplay a difficult stakeholder
- "briefing mode" — summarize Jeremy's tasks and context from memory

Current mode: {current_mode}
"""


@dataclass 
class VocalState:
    emotion: str
    confidence: float
    pitch_mean_hz: Optional[float]
    energy_rms: float
    speaking_rate_approx: float
    voiced_fraction: float


def format_vocal_state(state: VocalState) -> str:
    """Format vocal state as the [VOCAL_STATE] tag injected before each user message."""
    pitch_str = f"{state.pitch_mean_hz:.0f}hz" if state.pitch_mean_hz else "unknown"
    return (
        f"[VOCAL_STATE "
        f"emotion={state.emotion} "
        f"confidence={state.confidence:.2f} "
        f"pitch={pitch_str} "
        f"energy={state.energy_rms:.2f} "
        f"rate={state.speaking_rate_approx:.1f}ops "
        f"voiced={state.voiced_fraction:.2f}]"
    )


def build_user_message(transcript: str, vocal_state: Optional[VocalState]) -> str:
    """Combines transcript with vocal state tag."""
    if vocal_state and vocal_state.confidence > 0.3:
        tag = format_vocal_state(vocal_state)
        return f"{tag}\n{transcript}"
    return transcript


def build_system_prompt(
    baseline_context: str = "",
    session_history: str = "",
    current_mode: str = "coaching mode",
) -> str:
    return COACHING_SYSTEM_PROMPT.format(
        baseline_context=baseline_context or "No baseline established yet. First session: ask Jeremy to read a short neutral calibration script.",
        session_history=session_history or "No previous session history.",
        current_mode=current_mode,
    )

pipecat-agent/pipeline.py

"""
Main Pipecat pipeline assembly.
Connects LiveKit → Deepgram → SpeechBrain → Claude → ElevenLabs → LiveKit.
"""
import asyncio
import os
import logging
import httpx
from typing import Optional

from pipecat.frames.frames import (
    AudioRawFrame, TranscriptionFrame, TextFrame, EndFrame
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.anthropic import AnthropicLLMService
from pipecat.services.deepgram import DeepgramSTTService, LiveOptions
from pipecat.services.elevenlabs import ElevenLabsTTSService
from pipecat.transports.services.livekit import LiveKitTransport, LiveKitParams
from pipecat.processors.frame_processor import FrameProcessor, FrameDirection
from pipecat.vad.silero import SileroVADAnalyzer

from prompt_builder import build_system_prompt, build_user_message, VocalState

logger = logging.getLogger(__name__)

SPEECHBRAIN_URL = os.getenv("SPEECHBRAIN_SERVICE_URL", "http://speechbrain-service:8001")
ANTHROPIC_MODEL = "claude-sonnet-4-6"


class VocalStateInjector(FrameProcessor):
    """
    Intercepts TranscriptionFrames, fetches vocal analysis from SpeechBrain,
    and enriches the frame before passing to the LLM context.
    
    Audio buffer is maintained separately from the STT pipeline —
    SpeechBrain receives a copy of the raw audio for each utterance.
    """

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
        self._current_audio: list[bytes] = []
        self._http_client = httpx.AsyncClient(timeout=3.0)

    async def process_frame(self, frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        if isinstance(frame, AudioRawFrame):
            # Buffer audio for the current utterance
            self._current_audio.append(frame.audio)

        elif isinstance(frame, TranscriptionFrame):
            # Utterance complete — analyze buffered audio
            vocal_state = await self._analyze_buffered_audio()
            self._current_audio = []  # Reset buffer

            # Inject vocal state into the transcript
            enriched_text = build_user_message(frame.text, vocal_state)
            enriched_frame = TranscriptionFrame(
                text=enriched_text,
                user_id=frame.user_id,
                timestamp=frame.timestamp,
            )
            await self.push_frame(enriched_frame, direction)
        else:
            await self.push_frame(frame, direction)

    async def _analyze_buffered_audio(self) -> Optional[VocalState]:
        if not self._current_audio:
            return None

        try:
            import io, wave
            audio_bytes = b"".join(self._current_audio)
            # Wrap raw PCM in WAV container for SpeechBrain
            buf = io.BytesIO()
            with wave.open(buf, "wb") as wf:
                wf.setnchannels(1)
                wf.setsampwidth(2)   # 16-bit
                wf.setframerate(16000)
                wf.writeframes(audio_bytes)
            wav_bytes = buf.getvalue()

            response = await self._http_client.post(
                f"{SPEECHBRAIN_URL}/analyze",
                files={"audio": ("utterance.wav", wav_bytes, "audio/wav")},
            )
            response.raise_for_status()
            data = response.json()

            return VocalState(
                emotion=data.get("emotion", "neutral"),
                confidence=data.get("emotion_confidence", 0.0),
                pitch_mean_hz=data.get("pitch_mean_hz"),
                energy_rms=data.get("energy_rms", 0.0),
                speaking_rate_approx=data.get("speaking_rate_approx", 0.0),
                voiced_fraction=data.get("voiced_fraction", 0.0),
            )
        except Exception as e:
            logger.warning(f"SpeechBrain analysis failed (continuing without): {e}")
            return None


async def run_voice_coach_pipeline(
    livekit_url: str,
    livekit_token: str,
    anthropic_key: str,
    deepgram_key: str,
    elevenlabs_key: str,
    elevenlabs_voice_id: str,
    session_context: dict,
):
    """
    Assemble and run the full voice coaching pipeline.
    
    session_context contains: baseline_context, session_history, current_mode
    """
    # Transport: LiveKit WebRTC
    transport = LiveKitTransport(
        url=livekit_url,
        token=livekit_token,
        room_name="voicecoach",
        params=LiveKitParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            vad_analyzer=SileroVADAnalyzer(),
        ),
    )

    # STT: Deepgram streaming with aggressive endpointing for noisy car
    stt = DeepgramSTTService(
        api_key=deepgram_key,
        live_options=LiveOptions(
            model="nova-2",
            language="en-US",
            endpointing=500,        # 500ms silence = end of utterance
            interim_results=False,
            smart_format=True,
            punctuate=True,
        ),
    )

    # Vocal state injection (SpeechBrain)
    vocal_injector = VocalStateInjector()

    # LLM: Claude
    llm = AnthropicLLMService(
        api_key=anthropic_key,
        model=ANTHROPIC_MODEL,
        max_tokens=512,
    )

    # System prompt with context
    system_prompt = build_system_prompt(
        baseline_context=session_context.get("baseline_context", ""),
        session_history=session_context.get("session_history", ""),
        current_mode=session_context.get("current_mode", "coaching mode"),
    )

    context = OpenAILLMContext(
        messages=[{"role": "system", "content": system_prompt}]
    )
    context_aggregator = llm.create_context_aggregator(context)

    # TTS: ElevenLabs streaming
    tts = ElevenLabsTTSService(
        api_key=elevenlabs_key,
        voice_id=elevenlabs_voice_id,
        model="eleven_turbo_v2_5",  # Lowest latency model
        optimize_streaming_latency=4,
    )

    # Assemble pipeline
    pipeline = Pipeline([
        transport.input(),
        stt,
        vocal_injector,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
    ])

    task = PipelineTask(
        pipeline,
        params=PipelineParams(allow_interruptions=True),
    )

    runner = PipelineRunner()

    @transport.event_handler("on_first_participant_joined")
    async def on_first_participant_joined(transport, participant):
        logger.info(f"Participant joined: {participant['identity']}")
        await task.queue_frames([
            TextFrame("VoiceCoach connected. How are you doing today?")
        ])

    @transport.event_handler("on_participant_left")
    async def on_participant_left(transport, participant, reason):
        logger.info(f"Participant left: {participant['identity']}")
        await task.queue_frames([EndFrame()])

    await runner.run(task)

pipecat-agent/main.py

"""
Entry point for the Pipecat VoiceCoach agent.
Reads secrets from Docker Swarm secret files.
"""
import asyncio
import logging
import os
from pathlib import Path

import structlog
from livekit import api as livekit_api

from pipeline import run_voice_coach_pipeline
from openclaw_client import OpenClawClient

logging.basicConfig(level=logging.INFO)
logger = structlog.get_logger()


def read_secret(name: str, env_fallback: str = None) -> str:
    """Read from Docker Swarm secret file, fall back to env var."""
    secret_path = Path(f"/run/secrets/{name}")
    if secret_path.exists():
        return secret_path.read_text().strip()
    if env_fallback and os.getenv(env_fallback):
        return os.getenv(env_fallback)
    raise RuntimeError(f"Secret '{name}' not found at {secret_path} and no env fallback")


async def main():
    # Load secrets
    anthropic_key       = read_secret("voicecoach_anthropic_key", "ANTHROPIC_API_KEY")
    deepgram_key        = read_secret("voicecoach_deepgram_key", "DEEPGRAM_API_KEY")
    elevenlabs_key      = read_secret("voicecoach_elevenlabs_key", "ELEVENLABS_API_KEY")
    elevenlabs_voice_id = read_secret("voicecoach_elevenlabs_voice_id", "ELEVENLABS_VOICE_ID")
    livekit_api_key     = read_secret("voicecoach_livekit_api_key", "LIVEKIT_API_KEY")
    livekit_api_secret  = read_secret("voicecoach_livekit_api_secret", "LIVEKIT_API_SECRET")
    openclaw_token      = read_secret("voicecoach_openclaw_token", "OPENCLAW_TOKEN")

    livekit_url = os.getenv("LIVEKIT_URL", "ws://livekit-server:7880")
    openclaw_url = os.getenv("OPENCLAW_URL", "http://openclaw-voicecoach:3000")

    # Fetch session context from OpenClaw
    openclaw = OpenClawClient(openclaw_url, openclaw_token)
    session_context = await openclaw.get_session_context()
    logger.info("Session context loaded", mode=session_context.get("current_mode"))

    # Generate LiveKit access token for this agent
    token = livekit_api.AccessToken(livekit_api_key, livekit_api_secret)
    token.with_grants(livekit_api.VideoGrants(room_join=True, room="voicecoach"))
    token.identity = "voicecoach-agent"
    livekit_token = token.to_jwt()

    logger.info("Starting VoiceCoach pipeline", livekit_url=livekit_url)

    await run_voice_coach_pipeline(
        livekit_url=livekit_url,
        livekit_token=livekit_token,
        anthropic_key=anthropic_key,
        deepgram_key=deepgram_key,
        elevenlabs_key=elevenlabs_key,
        elevenlabs_voice_id=elevenlabs_voice_id,
        session_context=session_context,
    )


if __name__ == "__main__":
    asyncio.run(main())

pipecat-agent/openclaw_client.py

"""
Client for reading/writing session data to the OpenClaw VoiceCoach agent.
"""
import httpx
import logging

logger = logging.getLogger(__name__)


class OpenClawClient:
    def __init__(self, base_url: str, token: str):
        self.base_url = base_url.rstrip("/")
        self.headers = {"Authorization": f"Bearer {token}"}

    async def get_session_context(self) -> dict:
        """Fetch baseline and recent session history from OpenClaw memory."""
        try:
            async with httpx.AsyncClient(timeout=5.0) as client:
                r = await client.get(
                    f"{self.base_url}/api/memory",
                    headers=self.headers,
                    params={"keys": "vocal_baseline,recent_sessions,current_mode"},
                )
                r.raise_for_status()
                memory = r.json()

                return {
                    "baseline_context": memory.get("vocal_baseline", ""),
                    "session_history": memory.get("recent_sessions", ""),
                    "current_mode": memory.get("current_mode", "coaching mode"),
                }
        except Exception as e:
            logger.warning(f"Could not fetch OpenClaw context: {e}. Using defaults.")
            return {"baseline_context": "", "session_history": "", "current_mode": "coaching mode"}

    async def write_session_summary(self, summary: dict) -> bool:
        """Write end-of-session summary to OpenClaw memory."""
        try:
            async with httpx.AsyncClient(timeout=5.0) as client:
                r = await client.post(
                    f"{self.base_url}/api/memory",
                    headers=self.headers,
                    json={"key": "recent_sessions", "value": summary},
                )
                r.raise_for_status()
                return True
        except Exception as e:
            logger.error(f"Failed to write session summary: {e}")
            return False

pipecat-agent/Dockerfile

FROM python:3.11-slim

RUN apt-get update && apt-get install -y \
    libsndfile1 \
    ffmpeg \
    && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

HEALTHCHECK --interval=30s --timeout=10s --start-period=30s \
    CMD python -c "import asyncio; print('pipeline process alive')" || exit 1

CMD ["python", "main.py"]

pipecat-agent/tests/test_prompt_builder.py

"""Tests for prompt assembly."""
from prompt_builder import build_user_message, VocalState, build_system_prompt


def test_vocal_state_injected_into_message():
    state = VocalState(
        emotion="ang",
        confidence=0.82,
        pitch_mean_hz=215.0,
        energy_rms=0.71,
        speaking_rate_approx=4.8,
        voiced_fraction=0.91,
    )
    result = build_user_message("I can't get my team to listen", state)
    assert "[VOCAL_STATE" in result
    assert "emotion=ang" in result
    assert "I can't get my team to listen" in result


def test_low_confidence_vocal_state_omitted():
    state = VocalState(
        emotion="ang",
        confidence=0.15,  # Below threshold
        pitch_mean_hz=None,
        energy_rms=0.0,
        speaking_rate_approx=0.0,
        voiced_fraction=0.0,
    )
    result = build_user_message("Hello", state)
    assert "[VOCAL_STATE" not in result


def test_none_vocal_state_handled():
    result = build_user_message("Hello", None)
    assert result == "Hello"


def test_system_prompt_contains_mode():
    prompt = build_system_prompt(current_mode="vent mode")
    assert "vent mode" in prompt


def test_system_prompt_contains_baseline_when_provided():
    prompt = build_system_prompt(baseline_context="Jeremy's baseline pitch: 180hz")
    assert "180hz" in prompt

---

Step 4: LiveKit Server Config

livekit/livekit.yaml

port: 7880
rtc:
  tcp_port: 7881
  port_range_start: 50000
  port_range_end: 60000
  use_external_ip: true  # Required for remote clients (phone from road)

# Logging
logging:
  json: false
  level: info

# Room settings
room:
  max_participants: 5
  empty_timeout: 300    # Close room after 5min of no participants
  departure_timeout: 30

# Keys are loaded from environment — injected via Docker secrets
# Format: key_name: secret_value  (loaded from LIVEKIT_KEYS env var)

---

Step 5: Web Client

web-client/index.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
  <title>VoiceCoach</title>
  <link rel="stylesheet" href="style.css">
</head>
<body>
  <div class="container">
    <div class="header">
      <h1>🎙 VoiceCoach</h1>
      <div id="status" class="status disconnected">Disconnected</div>
    </div>

    <div class="controls">
      <button id="connectBtn" class="btn btn-primary">Start Session</button>
      <button id="disconnectBtn" class="btn btn-danger" style="display:none">End Session</button>
    </div>

    <div id="modeDisplay" class="mode-display">Mode: coaching</div>

    <div id="transcript" class="transcript">
      <p class="hint">Press "Start Session" to connect your voice.</p>
    </div>

    <div class="mode-buttons">
      <button class="mode-btn" onclick="setMode('coaching mode')">🎯 Coaching</button>
      <button class="mode-btn" onclick="setMode('vent mode')">💬 Vent</button>
      <button class="mode-btn" onclick="setMode('practice mode')">🎭 Practice</button>
      <button class="mode-btn" onclick="setMode('briefing mode')">📋 Briefing</button>
    </div>
  </div>

  <script src="https://cdn.jsdelivr.net/npm/livekit-client/dist/livekit-client.umd.min.js"></script>
  <script src="app.js"></script>
</body>
</html>

web-client/app.js

const GATEWAY_URL = window.location.origin + "/api";
let room = null;

async function getToken() {
  const response = await fetch(`${GATEWAY_URL}/token`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ identity: "jeremy-" + Date.now() }),
  });
  if (!response.ok) throw new Error("Failed to get token");
  return response.json();
}

async function connect() {
  setStatus("connecting", "Connecting...");
  try {
    const { token, url } = await getToken();
    room = new LivekitClient.Room({
      audioCaptureDefaults: {
        echoCancellation: true,
        noiseSuppression: true,
        autoGainControl: true,
      },
    });

    room.on(LivekitClient.RoomEvent.Connected, () => {
      setStatus("connected", "Connected");
      document.getElementById("connectBtn").style.display = "none";
      document.getElementById("disconnectBtn").style.display = "block";
    });

    room.on(LivekitClient.RoomEvent.Disconnected, () => {
      setStatus("disconnected", "Disconnected");
      document.getElementById("connectBtn").style.display = "block";
      document.getElementById("disconnectBtn").style.display = "none";
    });

    room.on(LivekitClient.RoomEvent.TrackSubscribed, (track) => {
      if (track.kind === LivekitClient.Track.Kind.Audio) {
        const element = track.attach();
        document.body.appendChild(element);
        element.play();
      }
    });

    await room.connect(url, token);
    await room.localParticipant.setMicrophoneEnabled(true);
  } catch (err) {
    setStatus("disconnected", "Error: " + err.message);
    console.error(err);
  }
}

async function disconnect() {
  if (room) {
    await room.disconnect();
    room = null;
  }
}

function setMode(mode) {
  document.getElementById("modeDisplay").textContent = "Mode: " + mode;
  if (room && room.localParticipant) {
    // Send mode as data message to agent
    const encoder = new TextEncoder();
    room.localParticipant.publishData(
      encoder.encode(JSON.stringify({ type: "mode_change", mode })),
      { reliable: true }
    );
  }
}

function setStatus(state, text) {
  const el = document.getElementById("status");
  el.className = "status " + state;
  el.textContent = text;
}

document.getElementById("connectBtn").addEventListener("click", connect);
document.getElementById("disconnectBtn").addEventListener("click", disconnect);

web-client/Dockerfile

FROM nginx:alpine

COPY index.html /usr/share/nginx/html/
COPY app.js /usr/share/nginx/html/
COPY style.css /usr/share/nginx/html/

# Token endpoint proxy config
COPY nginx.conf /etc/nginx/conf.d/default.conf

EXPOSE 80

web-client/nginx.conf

server {
    listen 80;
    root /usr/share/nginx/html;
    index index.html;

    # Proxy /api/* to the pipecat agent's token service
    location /api/ {
        proxy_pass http://pipecat-agent:8080/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
    }

    location / {
        try_files $uri $uri/ /index.html;
    }
}

---

Step 6: OpenClaw Agent Config

openclaw-agent/openclaw.json

{
  "name": "VoiceCoach",
  "description": "Real-time voice coaching agent for commute sessions. Integrates vocal analysis with communication coaching.",
  "version": "1.0.0",
  "model": {
    "provider": "anthropic",
    "name": "claude-sonnet-4-6",
    "temperature": 0.7,
    "max_tokens": 512
  },
  "memory": {
    "enabled": true,
    "keys": [
      "vocal_baseline",
      "recent_sessions",
      "current_mode",
      "action_items",
      "coaching_themes"
    ]
  },
  "communication": {
    "text": {
      "enabled": true
    },
    "voice": {
      "enabled": false
    }
  },
  "plugins": {
    "entries": {}
  },
  "api": {
    "enabled": true,
    "port": 3000,
    "cors": ["http://pipecat-agent:8080"]
  }
}

---

Step 7: Docker Stack Definition

docker-stack.yml

version: "3.8"

networks:
  hobbs-overlay:
    external: true
  voicecoach-internal:
    driver: overlay
    internal: true   # Not routable externally

volumes:
  speechbrain-models:
    driver: local
  openclaw-voicecoach-data:
    driver: local
  livekit-data:
    driver: local

secrets:
  voicecoach_anthropic_key:
    external: true
  voicecoach_deepgram_key:
    external: true
  voicecoach_elevenlabs_key:
    external: true
  voicecoach_elevenlabs_voice_id:
    external: true
  voicecoach_openclaw_token:
    external: true
  voicecoach_livekit_api_key:
    external: true
  voicecoach_livekit_api_secret:
    external: true

services:

  # ──────────────────────────────────────────────────────────
  # LiveKit WebRTC Server
  # ──────────────────────────────────────────────────────────
  livekit-server:
    image: livekit/livekit-server:latest
    command: --config /etc/livekit/livekit.yaml
    ports:
      - "7880:7880"       # HTTP/WS API
      - "7881:7881"       # RTC over TCP
      - "50000-60000:50000-60000/udp"  # WebRTC media
    volumes:
      - ./livekit/livekit.yaml:/etc/livekit/livekit.yaml:ro
      - livekit-data:/data
    environment:
      - LIVEKIT_KEYS_FILE=/run/secrets/voicecoach_livekit_api_key
    secrets:
      - voicecoach_livekit_api_key
      - voicecoach_livekit_api_secret
    networks:
      - hobbs-overlay
      - voicecoach-internal
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager   # LiveKit benefits from stable placement
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.livekit.rule=Host(`voice.hobbs.farm`)"
        - "traefik.http.routers.livekit.entrypoints=websecure"
        - "traefik.http.routers.livekit.tls.certresolver=cloudflare"
        - "traefik.http.services.livekit.loadbalancer.server.port=7880"
        - "traefik.http.routers.livekit.middlewares=livekit-websocket"
        - "traefik.http.middlewares.livekit-websocket.headers.customrequestheaders.Upgrade=websocket"
      restart_policy:
        condition: on-failure

  # ──────────────────────────────────────────────────────────
  # SpeechBrain Emotion Analysis Microservice (INTERNAL ONLY)
  # ──────────────────────────────────────────────────────────
  speechbrain-service:
    image: hobbs.farm/voicecoach/speechbrain:latest
    build:
      context: ./speechbrain-service
    volumes:
      - speechbrain-models:/app/models
    environment:
      - MODEL_DIR=/app/models
    networks:
      - voicecoach-internal
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.gpu != true   # Keep off GPU node unless configured
      resources:
        limits:
          cpus: "2"
          memory: "3G"
        reservations:
          memory: "2G"
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "python", "-c", "import httpx; httpx.get('http://localhost:8001/health').raise_for_status()"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 90s   # Model loading takes time

  # ──────────────────────────────────────────────────────────
  # Pipecat Voice Pipeline Agent
  # ──────────────────────────────────────────────────────────
  pipecat-agent:
    image: hobbs.farm/voicecoach/pipecat:latest
    build:
      context: ./pipecat-agent
    environment:
      - LIVEKIT_URL=ws://livekit-server:7880
      - SPEECHBRAIN_SERVICE_URL=http://speechbrain-service:8001
      - OPENCLAW_URL=http://openclaw-voicecoach:3000
    secrets:
      - voicecoach_anthropic_key
      - voicecoach_deepgram_key
      - voicecoach_elevenlabs_key
      - voicecoach_elevenlabs_voice_id
      - voicecoach_livekit_api_key
      - voicecoach_livekit_api_secret
      - voicecoach_openclaw_token
    networks:
      - voicecoach-internal
      - hobbs-overlay    # Needs internet for Anthropic/Deepgram/ElevenLabs APIs
    depends_on:
      - livekit-server
      - speechbrain-service
      - openclaw-voicecoach
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 10s

  # ──────────────────────────────────────────────────────────
  # OpenClaw VoiceCoach Agent
  # ──────────────────────────────────────────────────────────
  openclaw-voicecoach:
    image: ghcr.io/openclaw/openclaw:latest
    volumes:
      - ./openclaw-agent/openclaw.json:/app/openclaw.json:ro
      - ./openclaw-agent/system_prompt.md:/app/system_prompt.md:ro
      - ./openclaw-agent/skills:/app/skills:ro
      - openclaw-voicecoach-data:/app/data
    secrets:
      - voicecoach_anthropic_key
      - voicecoach_openclaw_token
    environment:
      - OPENCLAW_CONFIG=/app/openclaw.json
      - ANTHROPIC_API_KEY_FILE=/run/secrets/voicecoach_anthropic_key
    networks:
      - voicecoach-internal
      - hobbs-overlay
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3

  # ──────────────────────────────────────────────────────────
  # Web Client (phone interface)
  # ──────────────────────────────────────────────────────────
  voicecoach-web:
    image: hobbs.farm/voicecoach/web:latest
    build:
      context: ./web-client
    networks:
      - hobbs-overlay
      - voicecoach-internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.voicecoach.rule=Host(`coach.hobbs.farm`)"
        - "traefik.http.routers.voicecoach.entrypoints=websecure"
        - "traefik.http.routers.voicecoach.tls.certresolver=cloudflare"
        - "traefik.http.services.voicecoach.loadbalancer.server.port=80"
        - "traefik.http.routers.voicecoach.middlewares=voicecoach-auth"
        - "traefik.http.middlewares.voicecoach-auth.basicauth.users=jeremy:$$apr1$$REPLACEME"
      restart_policy:
        condition: on-failure

Note for Claude Code: Replace $$apr1$$REPLACEME in the basicauth label with the output of:

htpasswd -nb jeremy yourpassword | sed 's/\$/\$\$/g'

---

Step 8: Makefile

Makefile

.PHONY: build deploy secrets test smoke e2e clean logs

# Build all images
build:
	docker build -t hobbs.farm/voicecoach/speechbrain:latest ./speechbrain-service
	docker build -t hobbs.farm/voicecoach/pipecat:latest ./pipecat-agent
	docker build -t hobbs.farm/voicecoach/web:latest ./web-client

# Create secrets (run once)
secrets:
	bash scripts/create-secrets.sh

# Deploy stack
deploy: build
	docker stack deploy -c docker-stack.yml voicecoach

# Run unit tests (local, no Docker required)
test:
	cd speechbrain-service && python -m pytest tests/ -v
	cd pipecat-agent && python -m pytest tests/ -v

# Quick smoke test after deploy
smoke:
	bash scripts/smoke-test.sh

# Full end-to-end test
e2e:
	bash scripts/e2e-test.sh

# Follow all service logs
logs:
	docker service logs voicecoach_pipecat-agent -f

# Remove stack
clean:
	docker stack rm voicecoach

---

Step 9: Test & Validation Scripts

scripts/smoke-test.sh

#!/bin/bash
# Quick validation — run immediately after deploy
set -e

echo "=== VoiceCoach Smoke Tests ==="

# 1. Check all services running
echo ""
echo "→ Checking service replicas..."
docker service ls | grep voicecoach
FAILED=$(docker service ls | grep voicecoach | grep "0/" | wc -l)
if [ "$FAILED" -gt 0 ]; then
  echo "❌ Some services have 0 replicas. Check: docker service ps voicecoach_<name>"
  exit 1
fi
echo "✅ All services running"

# 2. LiveKit health
echo ""
echo "→ LiveKit health check..."
curl -sf https://voice.hobbs.farm/rtc/health > /dev/null && echo "✅ LiveKit responding" || echo "❌ LiveKit not responding"

# 3. SpeechBrain health (via internal overlay — run from swarm manager)
echo ""
echo "→ SpeechBrain health check (internal)..."
# Run from a container on the voicecoach-internal network
docker run --rm --network voicecoach_voicecoach-internal \
  curlimages/curl:latest \
  curl -sf http://speechbrain-service:8001/health && echo "✅ SpeechBrain responding" || echo "❌ SpeechBrain not responding"

# 4. Web client
echo ""
echo "→ Web client check..."
curl -sf -u jeremy:yourpassword https://coach.hobbs.farm > /dev/null && echo "✅ Web client responding" || echo "❌ Web client not responding"

# 5. OpenClaw health
echo ""
echo "→ OpenClaw health check (internal)..."
docker run --rm --network voicecoach_voicecoach-internal \
  curlimages/curl:latest \
  curl -sf http://openclaw-voicecoach:3000/health && echo "✅ OpenClaw responding" || echo "❌ OpenClaw not responding"

echo ""
echo "=== Smoke tests complete ==="

scripts/e2e-test.sh

#!/bin/bash
# Full end-to-end test using a pre-recorded test WAV
# Requires: livekit-cli installed (go install github.com/livekit/livekit-cli/cmd/lk@latest)

echo "=== VoiceCoach End-to-End Test ==="
echo "This test joins the LiveKit room as a test participant and"
echo "sends a pre-recorded audio clip, then checks for TTS output."

LK_KEY=$(docker secret inspect voicecoach_livekit_api_key --format '{{.Spec.Name}}' 2>/dev/null || echo "not-found")

echo ""
echo "→ Downloading test audio..."
curl -sL "https://www2.cs.uic.edu/~i101/SoundFiles/StarWars60.wav" -o /tmp/test_audio.wav

echo ""
echo "→ Joining LiveKit room and sending test audio..."
echo "  (Manual step: open https://coach.hobbs.farm on phone and verify you hear a response)"
echo ""
echo "→ Alternatively, use livekit-cli:"
echo "  lk room join --url wss://voice.hobbs.farm --api-key \$KEY --api-secret \$SECRET voicecoach"
echo ""

echo "=== E2E test requires manual verification at this time ==="
echo "→ Automated e2e test with audio injection coming in v1.1"

tests/test_latency.sh

#!/bin/bash
# Measure SpeechBrain analysis latency
echo "=== Latency Test: SpeechBrain Analysis ==="

# Download a 2-second speech sample
curl -sL "https://www2.cs.uic.edu/~i101/SoundFiles/preamble.wav" -o /tmp/latency_test.wav

# Test via internal network
START=$(date +%s%3N)
docker run --rm --network voicecoach_voicecoach-internal \
  -v /tmp/latency_test.wav:/test.wav \
  curlimages/curl:latest \
  curl -sf -X POST http://speechbrain-service:8001/analyze \
    -F "audio=@/test.wav" > /tmp/latency_result.json
END=$(date +%s%3N)

ELAPSED=$((END - START))
echo "Analysis latency: ${ELAPSED}ms"

if [ "$ELAPSED" -lt 3000 ]; then
  echo "✅ Within 3s budget"
else
  echo "⚠️  Exceeds 3s budget — consider GPU or lighter model"
fi

echo ""
cat /tmp/latency_result.json | python3 -m json.tool

---

Step 10: Deploy Order

Claude Code should execute in this exact order:

# 1. Clone/create repository structure
# (Claude Code: create all files from this document)

# 2. Create Docker Swarm secrets (ONCE)
make secrets

# 3. Confirm overlay network
docker network ls | grep hobbs-overlay || docker network create --driver overlay --attachable hobbs-overlay

# 4. Build images
make build

# 5. Deploy stack
make deploy

# 6. Monitor startup (SpeechBrain takes ~90s to load model)
watch docker service ls

# 7. Run smoke tests
make smoke

# 8. Run unit tests
make test

# 9. Open coach.hobbs.farm on phone and test voice

---

Interface Change Summary

What changes from your current Matrix setup:

Capability	Before	After
Real-time voice with agents	❌ Not supported	✅ coach.hobbs.farm (phone browser)
Text chat with Pluto/agents	✅ Matrix stays	✅ Matrix stays (unchanged)
Async task dispatch	✅ Matrix/Dispatch	✅ Matrix/Dispatch stays
Vocal analysis coaching	❌ Not possible	✅ Full pipeline
Session memory	✅ OpenClaw memory	✅ OpenClaw memory (enhanced)

Matrix is NOT replaced. It remains your primary async interface for Pluto and all text-based agent interaction. VoiceCoach is additive — a dedicated real-time voice interface for commute sessions only.

---

Rollback

# Remove the voice coach stack entirely (OpenClaw memory data preserved in volume)
docker stack rm voicecoach

# Remove secrets if rotating
docker secret rm voicecoach_anthropic_key voicecoach_deepgram_key \
  voicecoach_elevenlabs_key voicecoach_elevenlabs_voice_id \
  voicecoach_openclaw_token voicecoach_livekit_api_key voicecoach_livekit_api_secret

---

Known Issues & Troubleshooting

SpeechBrain takes too long on CPU:

• First inference is always slower (model warmup). Subsequent calls are faster.
• If consistently > 3s: add a lighter model path using librosa-only analysis (no SpeechBrain) as fallback.

WebRTC media not flowing (hear nothing):

• Almost certainly firewall/NAT. Confirm UDP 50000-60000 is open at router.
• LiveKit logs: docker service logs voicecoach_livekit-server -f

Deepgram returning no transcripts:

• Check API key secret. Deepgram errors appear in pipecat-agent logs.
• Try endpointing=1000 if utterances are being cut off.

ElevenLabs high latency:

• Switch to eleven_turbo_v2_5 model (already set). If still slow, use edge-tts locally.
• Set optimize_streaming_latency=4 (already set — maximum optimization).

SpeechBrain emotion always "neutral":

• Normal for the IEMOCAP model on non-actor speech. Acoustic features (pitch, energy, rate) are more reliable than the emotion label for real speech. Rely on those for coaching signals.