hobbs-iac/VOICECOACH_V2_DEPLOY.md

VoiceCoach v2 — Fully Self-Hosted (Except Claude API)

OpenClaw as LLM Gateway + Whisper STT + Kokoro TTS

Supersedes: VOICECOACH_DEPLOY.md | For use with Claude Code

---

What Changed and Why

v1 architecture problems:

• Pipecat called Claude API directly — OpenClaw was just a memory store bolted on the side
• Deepgram (external, paid per-minute) handled STT
• ElevenLabs (external, paid per-character) handled TTS
• OpenClaw's tools, skills, memory, and agent routing were bypassed entirely during live conversation

v2 architecture:

• Pipecat routes ALL LLM calls through the OpenClaw gateway via its built-in OpenAI-compatible endpoint
• OpenClaw handles the conversation, calls Claude API, applies its full tool/memory/skill stack
• STT: Speaches container (self-hosted faster-whisper) — no API key, no per-minute cost
• TTS: Kokoro via Speaches (self-hosted, 82M param model) — no API key, no per-character cost
• Only external call remaining: Claude API via OpenClaw

The key insight: OpenClaw's gateway exposes /v1/chat/completions. Setting model: "openclaw/voicecoach" routes to the VoiceCoach agent specifically — same codepath as any other OpenClaw session, so memory, skills, tools, and the vocal coaching system prompt all apply natively.

---

Revised Architecture

┌─────────────────────────────────────────────────────────┐
│                    PHONE (in truck)                      │
│         coach.hobbs.farm — browser WebRTC               │
└──────────────────────┬──────────────────────────────────┘
                       │ WebRTC (DTLS/SRTP)
                       ▼
┌─────────────────────────────────────────────────────────┐
│              LiveKit Server (voice.hobbs.farm)           │
│              Self-hosted WebRTC SFU                      │
└──────────┬──────────────────────────────────────────────┘
           │
           ▼
┌──────────────────────────────────────────────────────────┐
│                  Pipecat Pipeline Agent                   │
│                                                           │
│  ┌───────────────┐    ┌─────────────────┐                │
│  │ Speaches STT  │    │  VocalState      │                │
│  │ (Whisper)     │───▶│  Injector        │                │
│  │ self-hosted   │    │  (SpeechBrain)   │                │
│  └───────────────┘    └────────┬────────┘                │
│                                │                          │
│                                ▼                          │
│                   ┌────────────────────────┐             │
│                   │  OpenClaw Gateway      │             │
│                   │  /v1/chat/completions  │             │
│                   │  model=openclaw/       │             │
│                   │    voicecoach          │             │
│                   └──────────┬─────────────┘             │
│                              │ (Claude API — only        │
│                              │  external call)           │
│                              ▼                            │
│                   ┌────────────────────────┐             │
│                   │  Speaches TTS          │             │
│                   │  (Kokoro 82M)          │             │
│                   │  self-hosted           │             │
│                   └──────────┬─────────────┘             │
│                              │ audio                     │
└──────────────────────────────┼──────────────────────────┘
                               │ audio frames → LiveKit
                               ▼
                         Phone speaker


┌──────────────────────────────────────────────────────────┐
│            OpenClaw VoiceCoach Agent                      │
│                                                           │
│  - Receives ALL conversation turns from Pipecat          │
│  - Applies system prompt + coaching skills               │
│  - Reads/writes session memory natively                  │
│  - Calls Claude API (anthropic/claude-sonnet-4-6)        │
│  - Tools available: memory, search, Pluto integration    │
│  - Exposes: openclaw-voicecoach:3000/v1/chat/completions │
└──────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────┐
│            Speaches (STT + TTS)                           │
│  OpenAI-compatible API for both Whisper and Kokoro        │
│  POST /v1/audio/transcriptions  (Whisper large-v3-turbo) │
│  POST /v1/audio/speech          (Kokoro 82M)             │
│  Internal network only — no external exposure            │
└──────────────────────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────┐
│            SpeechBrain (unchanged from v1)                │
│  POST /analyze — vocal emotion + prosody features        │
└──────────────────────────────────────────────────────────┘

---

Service Inventory

Service	Image	External?	Port (internal)
livekit-server	livekit/livekit-server	UDP 50000-60000 (WebRTC only)	7880
speaches	fedirz/faster-whisper-server	❌ internal only	8000
speechbrain-service	hobbs.farm/voicecoach/speechbrain	❌ internal only	8001
openclaw-voicecoach	ghcr.io/openclaw/openclaw	❌ internal only	3000
pipecat-agent	hobbs.farm/voicecoach/pipecat	❌ internal only	—
voicecoach-web	hobbs.farm/voicecoach/web	✅ coach.hobbs.farm	80

External API calls: Claude API only (via OpenClaw)

Secrets needed: Anthropic API key, LiveKit key pair, OpenClaw token Removed secrets: Deepgram key, ElevenLabs key, ElevenLabs voice ID

---

What Claude Code Should Do With the v1 Files

KEEP unchanged:
  speechbrain-service/        (no changes)
  livekit/                    (no changes)
  web-client/                 (no changes)
  scripts/create-secrets.sh   (remove Deepgram + ElevenLabs steps)

REPLACE entirely:
  pipecat-agent/pipeline.py
  pipecat-agent/main.py
  pipecat-agent/requirements.txt
  openclaw-agent/openclaw.json
  openclaw-agent/system_prompt.md
  docker-stack.yml

ADD new:
  speaches/                   (new service)

---

Step 1: Updated Secrets

Remove Deepgram and ElevenLabs. Keep only:

#!/bin/bash
# scripts/create-secrets.sh (v2 — trimmed)
set -e

echo "Creating Docker Swarm secrets for VoiceCoach v2..."

read -s -p "Anthropic API Key: " ANTHROPIC_KEY && echo
read -s -p "OpenClaw Agent Token: " OPENCLAW_TOKEN && echo

LIVEKIT_API_KEY="voicecoach-$(openssl rand -hex 4)"
LIVEKIT_API_SECRET=$(openssl rand -base64 32)

echo "$ANTHROPIC_KEY"      | docker secret create voicecoach_anthropic_key -
echo "$OPENCLAW_TOKEN"     | docker secret create voicecoach_openclaw_token -
echo "$LIVEKIT_API_KEY"    | docker secret create voicecoach_livekit_api_key -
echo "$LIVEKIT_API_SECRET" | docker secret create voicecoach_livekit_api_secret -

echo ""
echo "✅ Secrets created."
echo "   LiveKit API Key:    $LIVEKIT_API_KEY"
echo "   LiveKit API Secret: $LIVEKIT_API_SECRET"
echo "   Save these — they cannot be retrieved from Swarm secrets."

---

Step 2: Speaches (Self-Hosted STT + TTS)

Speaches is a single container that serves both Whisper (STT) and Kokoro (TTS) via an OpenAI-compatible API. No configuration files needed — environment variables control model selection.

No Dockerfile needed — use the official image directly in the stack.

The Speaches service in docker-stack.yml:

  speaches:
    image: fedirz/faster-whisper-server:latest-cpu
    # Use :latest-cuda if a GPU node is available — significantly faster
    environment:
      - WHISPER__MODEL=large-v3-turbo       # Best speed/accuracy balance
      - WHISPER__INFERENCE_DEVICE=cpu        # Change to cuda if GPU available
      - WHISPER__COMPUTE_TYPE=int8           # Quantized for CPU efficiency
      - TTS__MODEL=kokoro                    # Kokoro 82M
      - TTS__DEFAULT_VOICE=af_nova           # American female — natural coaching voice
                                             # Other options: af_sky, am_adam, bf_emma
      - TTS__LANGUAGE=en-us
    volumes:
      - speaches-models:/root/.cache         # Cache downloaded models
    networks:
      - voicecoach-internal
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "3"
          memory: "4G"
        reservations:
          memory: "2G"
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s    # Kokoro + Whisper model download on first start

First-start note: Speaches downloads Whisper large-v3-turbo (~1.5GB) and Kokoro (~330MB) on first boot. Subsequent starts use the cached volume. Watch logs: docker service logs voicecoach_speaches -f

Test Speaches manually after deploy:

# Test STT (from inside overlay network)
docker run --rm --network voicecoach_voicecoach-internal \
  -v /tmp/test.wav:/test.wav \
  curlimages/curl:latest \
  curl -s -X POST http://speaches:8000/v1/audio/transcriptions \
    -F "file=@/test.wav" \
    -F "model=Systran/faster-whisper-large-v3-turbo"

# Test TTS
docker run --rm --network voicecoach_voicecoach-internal \
  curlimages/curl:latest \
  curl -s -X POST http://speaches:8000/v1/audio/speech \
    -H "Content-Type: application/json" \
    -d '{"model":"kokoro","input":"Testing VoiceCoach TTS","voice":"af_nova"}' \
    -o /dev/null && echo "TTS OK"

---

Step 3: Updated OpenClaw Agent Config

The key changes:

1. OpenAI-compatible gateway endpoint enabled
2. Anthropic as the LLM provider
3. System prompt is the full coaching persona (not just memory context)
4. Memory keys expanded for vocal baseline

openclaw-agent/openclaw.json

{
  "name": "VoiceCoach",
  "description": "Real-time voice coaching agent. Receives live conversation turns from the Pipecat pipeline via the OpenAI-compatible gateway endpoint.",
  "version": "2.0.0",
  "model": {
    "provider": "anthropic",
    "name": "claude-sonnet-4-6",
    "temperature": 0.7,
    "max_tokens": 512
  },
  "gateway": {
    "openaiApi": {
      "enabled": true,
      "agents": ["voicecoach"]
    },
    "auth": {
      "token": "${OPENCLAW_TOKEN}"
    }
  },
  "memory": {
    "enabled": true,
    "keys": [
      "vocal_baseline",
      "recent_sessions",
      "current_mode",
      "action_items",
      "coaching_themes",
      "jeremy_context"
    ]
  },
  "systemPrompt": {
    "file": "/app/system_prompt.md"
  },
  "communication": {
    "text": { "enabled": true },
    "voice": { "enabled": false }
  },
  "api": {
    "enabled": true,
    "port": 3000
  }
}

openclaw-agent/system_prompt.md

You are VoiceCoach, a personal communication and leadership coach for Jeremy.
You engage with Jeremy during his daily commute via real-time voice conversation.

## Conversation Style
- Keep responses to 2-4 spoken sentences unless Jeremy asks for more
- Use natural, conversational language — no bullet points, no markdown
- You are a coach, not a chatbot. Ask questions. Challenge thinking. Give direct feedback.

## Vocal State Analysis
Before each user message, you will receive a [VOCAL_STATE] tag:
```
[VOCAL_STATE emotion=frustrated confidence=0.82 pitch=215hz energy=0.71 rate=4.8ops voiced=0.91]
```

Use this to coach on HOW Jeremy communicates, not just WHAT he says:
- High pitch + high energy + fast rate = stressed, rushed — note this
- Low voiced fraction = hesitant, uncertain — worth exploring
- Elevated pitch from baseline = emotional arousal — name it gently
- Never quote the raw numbers. Translate to natural observations.
- Example: "I can hear some tension in how you're describing that" not "your pitch was 215hz"

Only comment on vocal state when it's meaningfully different from baseline or
when it's directly relevant to the coaching topic. Don't narrate every turn.

## Coaching Modes
Jeremy can switch modes mid-conversation. Honor the switch immediately.

**coaching mode** (default): Active feedback on both content and delivery.
  Ask probing questions. Notice patterns. Give direct recommendations.

**vent mode**: Listen and reflect. Minimal advice. Validate feelings.
  Mirror back what you hear. Let Jeremy process out loud.

**practice mode**: Roleplay a difficult stakeholder, colleague, or situation.
  Stay in character. Be realistic — not easy. Give feedback after.

**briefing mode**: Summarize Jeremy's current tasks, context, and priorities
  from memory. Be crisp. Flag anything time-sensitive.

## Memory Usage
You have access to:
- `vocal_baseline`: Jeremy's neutral voice characteristics for comparison
- `recent_sessions`: Last 3 session summaries with topics and vocal trends
- `action_items`: Commitments Jeremy made in previous sessions
- `coaching_themes`: Recurring themes across sessions (track these actively)
- `jeremy_context`: Jeremy's role, team, current challenges

At the end of each session, write a summary to `recent_sessions` including:
- Topics discussed
- Vocal patterns observed
- Action items for next session
- Any new coaching themes identified

## Baseline Calibration (First Session)
If `vocal_baseline` is empty, the first 60 seconds should be calibration.
Say: "Before we dive in, let's do a quick baseline. Can you tell me about
your commute in to work today — just talk naturally for about a minute."
Then write the averaged vocal readings to `vocal_baseline`.

## Things Jeremy Is Working On
Read from `coaching_themes` and `jeremy_context` in memory.
Reference previous sessions naturally: "Last week you mentioned..."
Track whether previous action items were attempted.

---

Step 4: Updated Pipecat Agent

The pipeline now uses:

• OpenAISTTService pointed at Speaches (Whisper)
• OpenAILLMService pointed at OpenClaw gateway
• OpenAITTSService pointed at Speaches (Kokoro)
• No external API keys in the pipeline itself

pipecat-agent/requirements.txt

pipecat-ai[livekit,silero,openai]==0.0.48
httpx==0.27.0
python-dotenv==1.0.1
structlog==24.4.0
livekit-api==0.7.0

pipecat-agent/pipeline.py

"""
Pipecat pipeline — v2 fully self-hosted.

STT:  Speaches (faster-whisper) via OpenAI-compatible API
LLM:  OpenClaw VoiceCoach agent via OpenAI-compatible gateway
TTS:  Speaches (Kokoro 82M) via OpenAI-compatible API
"""
import asyncio
import io
import json
import logging
import os
import uuid
import wave
from datetime import datetime
from pathlib import Path
from typing import Optional

import httpx
from pipecat.frames.frames import (
    AudioRawFrame, EndFrame, TextFrame, TranscriptionFrame
)
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineParams, PipelineTask
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.processors.frame_processor import FrameProcessor, FrameDirection
from pipecat.services.openai import OpenAILLMService, OpenAISTTService, OpenAITTSService
from pipecat.transports.services.livekit import LiveKitTransport, LiveKitParams
from pipecat.vad.silero import SileroVADAnalyzer

from prompt_builder import build_user_message, build_system_prompt, VocalState

logger = logging.getLogger(__name__)

# Internal service URLs — all on voicecoach-internal overlay network
SPEACHES_URL        = os.getenv("SPEACHES_URL",        "http://speaches:8000")
OPENCLAW_URL        = os.getenv("OPENCLAW_URL",         "http://openclaw-voicecoach:3000")
SPEECHBRAIN_URL     = os.getenv("SPEECHBRAIN_URL",      "http://speechbrain-service:8001")

WHISPER_MODEL       = os.getenv("WHISPER_MODEL",        "Systran/faster-whisper-large-v3-turbo")
KOKORO_VOICE        = os.getenv("KOKORO_VOICE",         "af_nova")
OPENCLAW_AGENT_ID   = os.getenv("OPENCLAW_AGENT_ID",    "voicecoach")

CLIPS_DIR           = Path(os.getenv("CLIPS_DIR",       "/data/clips/pending"))
UNCERTAINTY_THRESHOLD = 0.60
MIN_CLIP_DURATION   = 1.0


class VocalStateInjector(FrameProcessor):
    """
    Buffers audio per utterance, sends to SpeechBrain for analysis,
    enriches the TranscriptionFrame with [VOCAL_STATE] metadata,
    and saves clips for active labeling pipeline.
    """

    def __init__(self, session_id: str, **kwargs):
        super().__init__(**kwargs)
        self._session_id = session_id
        self._audio_buffer: list[bytes] = []
        self._http = httpx.AsyncClient(timeout=3.0)

    async def process_frame(self, frame, direction: FrameDirection):
        await super().process_frame(frame, direction)

        if isinstance(frame, AudioRawFrame):
            self._audio_buffer.append(frame.audio)

        elif isinstance(frame, TranscriptionFrame):
            wav_bytes = self._flush_audio_to_wav()
            self._audio_buffer = []

            vocal_state = None
            if wav_bytes:
                vocal_state, analysis = await self._analyze(wav_bytes)
                if analysis and self._clip_worth_saving(analysis, wav_bytes):
                    await self._save_clip(wav_bytes, analysis)

            enriched_text = build_user_message(frame.text, vocal_state)
            await self.push_frame(
                TranscriptionFrame(
                    text=enriched_text,
                    user_id=frame.user_id,
                    timestamp=frame.timestamp,
                ),
                direction,
            )
        else:
            await self.push_frame(frame, direction)

    def _flush_audio_to_wav(self) -> Optional[bytes]:
        if not self._audio_buffer:
            return None
        raw = b"".join(self._audio_buffer)
        buf = io.BytesIO()
        with wave.open(buf, "wb") as wf:
            wf.setnchannels(1)
            wf.setsampwidth(2)   # 16-bit PCM
            wf.setframerate(16000)
            wf.writeframes(raw)
        return buf.getvalue()

    async def _analyze(self, wav_bytes: bytes):
        try:
            r = await self._http.post(
                f"{SPEECHBRAIN_URL}/analyze",
                files={"audio": ("u.wav", wav_bytes, "audio/wav")},
            )
            r.raise_for_status()
            d = r.json()
            return (
                VocalState(
                    emotion=d.get("emotion", "neutral"),
                    confidence=d.get("emotion_confidence", 0.0),
                    pitch_mean_hz=d.get("pitch_mean_hz"),
                    energy_rms=d.get("energy_rms", 0.0),
                    speaking_rate_approx=d.get("speaking_rate_approx", 0.0),
                    voiced_fraction=d.get("voiced_fraction", 0.0),
                ),
                d,
            )
        except Exception as e:
            logger.warning(f"SpeechBrain unavailable: {e} — continuing without vocal state")
            return None, None

    def _clip_worth_saving(self, analysis: dict, wav_bytes: bytes) -> bool:
        duration = len(wav_bytes) / (16000 * 2)  # rough bytes-to-seconds
        return duration >= MIN_CLIP_DURATION

    async def _save_clip(self, wav_bytes: bytes, analysis: dict):
        try:
            CLIPS_DIR.mkdir(parents=True, exist_ok=True)
            clip_id = f"{self._session_id}_{uuid.uuid4().hex[:8]}"
            (CLIPS_DIR / f"{clip_id}.wav").write_bytes(wav_bytes)
            meta = {
                "clip_id": clip_id,
                "session_id": self._session_id,
                "timestamp": datetime.utcnow().isoformat(),
                "duration_seconds": len(wav_bytes) / (16000 * 2),
                "speechbrain_emotion": analysis.get("emotion"),
                "speechbrain_confidence": analysis.get("emotion_confidence"),
                "emotion_scores": analysis.get("emotion_scores", {}),
                "pitch_mean_hz": analysis.get("pitch_mean_hz"),
                "energy_rms": analysis.get("energy_rms"),
                "speaking_rate_approx": analysis.get("speaking_rate_approx"),
                "labeled": False,
                "confirmed_label": None,
                "uncertain": analysis.get("emotion_confidence", 1.0) < UNCERTAINTY_THRESHOLD,
            }
            (CLIPS_DIR / f"{clip_id}.json").write_text(json.dumps(meta, indent=2))
        except Exception as e:
            logger.warning(f"Failed to save clip: {e}")


async def run_voice_coach_pipeline(
    livekit_url: str,
    livekit_token: str,
    openclaw_token: str,
):
    session_id = datetime.utcnow().strftime("%Y%m%d-%H%M%S")

    # ── Transport ─────────────────────────────────────────────────────────────
    transport = LiveKitTransport(
        url=livekit_url,
        token=livekit_token,
        room_name="voicecoach",
        params=LiveKitParams(
            audio_in_enabled=True,
            audio_out_enabled=True,
            vad_analyzer=SileroVADAnalyzer(),
        ),
    )

    # ── STT: Speaches / faster-whisper ────────────────────────────────────────
    # OpenAISTTService accepts any OpenAI-compatible transcription endpoint
    stt = OpenAISTTService(
        api_key="speaches",           # Speaches doesn't require a real key
        base_url=f"{SPEACHES_URL}/v1",
        model=WHISPER_MODEL,
        language="en",
    )

    # ── Vocal State Injection ─────────────────────────────────────────────────
    vocal_injector = VocalStateInjector(session_id=session_id)

    # ── LLM: OpenClaw VoiceCoach agent via OpenAI-compatible gateway ──────────
    # OpenClaw's gateway exposes /v1/chat/completions
    # model="openclaw/voicecoach" routes to the VoiceCoach agent specifically
    # This means ALL OpenClaw features apply: memory, tools, skills, system prompt
    llm = OpenAILLMService(
        api_key=openclaw_token,
        base_url=f"{OPENCLAW_URL}/v1",
        model=f"openclaw/{OPENCLAW_AGENT_ID}",
        max_tokens=512,
    )

    # Context — system prompt is defined in OpenClaw's system_prompt.md
    # We pass a minimal context here; OpenClaw applies its full prompt server-side
    context = OpenAILLMContext(messages=[])
    context_aggregator = llm.create_context_aggregator(context)

    # ── TTS: Speaches / Kokoro ────────────────────────────────────────────────
    tts = OpenAITTSService(
        api_key="speaches",
        base_url=f"{SPEACHES_URL}/v1",
        model="kokoro",
        voice=KOKORO_VOICE,
    )

    # ── Pipeline Assembly ─────────────────────────────────────────────────────
    pipeline = Pipeline([
        transport.input(),
        stt,
        vocal_injector,
        context_aggregator.user(),
        llm,
        tts,
        transport.output(),
        context_aggregator.assistant(),
    ])

    task = PipelineTask(
        pipeline,
        params=PipelineParams(allow_interruptions=True),
    )
    runner = PipelineRunner()

    @transport.event_handler("on_first_participant_joined")
    async def on_first_participant_joined(transport, participant):
        logger.info(f"Participant joined: {participant['identity']}")
        # OpenClaw will handle the greeting based on its memory of previous sessions
        # Send a minimal trigger to start the conversation
        await task.queue_frames([
            TextFrame("session_start")
        ])

    @transport.event_handler("on_participant_left")
    async def on_participant_left(transport, participant, reason):
        logger.info(f"Session ending: {session_id}")
        await task.queue_frames([EndFrame()])
        # Trigger post-session labeling (from ACTIVE_LABELING.md)
        _trigger_labeler(session_id)

    await runner.run(task)


def _trigger_labeler(session_id: str):
    """Fire-and-forget: trigger session labeling container after session ends."""
    import subprocess
    try:
        subprocess.Popen([
            "docker", "run", "--rm",
            "--network", "voicecoach_voicecoach-internal",
            "--mount", "source=voicecoach-clips,target=/data/clips",
            "--secret", "voicecoach_matrix_token",
            "--secret", "voicecoach_matrix_bot_id",
            "--secret", "voicecoach_matrix_jeremy_id",
            "--secret", "voicecoach_matrix_homeserver",
            "hobbs.farm/voicecoach/session-labeler:latest",
            "python", "main.py", session_id,
        ])
        logger.info(f"Session labeler triggered for {session_id}")
    except Exception as e:
        logger.warning(f"Could not trigger labeler: {e}")

pipecat-agent/main.py

"""
VoiceCoach pipeline entry point — v2.
Reads secrets, generates LiveKit token, runs pipeline.
"""
import asyncio
import logging
import os
from pathlib import Path

import structlog
from livekit import api as livekit_api

from pipeline import run_voice_coach_pipeline

logging.basicConfig(level=logging.INFO)
logger = structlog.get_logger()


def read_secret(name: str, env_fallback: str = None) -> str:
    path = Path(f"/run/secrets/{name}")
    if path.exists():
        return path.read_text().strip()
    if env_fallback and os.getenv(env_fallback):
        return os.getenv(env_fallback)
    raise RuntimeError(f"Secret '{name}' not found")


async def main():
    livekit_api_key    = read_secret("voicecoach_livekit_api_key",    "LIVEKIT_API_KEY")
    livekit_api_secret = read_secret("voicecoach_livekit_api_secret", "LIVEKIT_API_SECRET")
    openclaw_token     = read_secret("voicecoach_openclaw_token",      "OPENCLAW_TOKEN")

    livekit_url = os.getenv("LIVEKIT_URL", "ws://livekit-server:7880")

    token = livekit_api.AccessToken(livekit_api_key, livekit_api_secret)
    token.with_grants(livekit_api.VideoGrants(room_join=True, room="voicecoach"))
    token.identity = "voicecoach-agent"
    livekit_token = token.to_jwt()

    logger.info("Starting VoiceCoach v2 pipeline", livekit_url=livekit_url)

    await run_voice_coach_pipeline(
        livekit_url=livekit_url,
        livekit_token=livekit_token,
        openclaw_token=openclaw_token,
    )


if __name__ == "__main__":
    asyncio.run(main())

pipecat-agent/Dockerfile

FROM python:3.11-slim

RUN apt-get update && apt-get install -y libsndfile1 ffmpeg && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

CMD ["python", "main.py"]

---

Step 5: Updated Tests

pipecat-agent/tests/test_pipeline.py

"""
Pipeline integration tests — v2.
Verifies routing through OpenClaw gateway and Speaches services.
"""
import pytest
from unittest.mock import AsyncMock, MagicMock, patch


def test_stt_uses_speaches_not_deepgram():
    """STT must be configured to use Speaches base URL, not Deepgram."""
    from pipecat.services.openai import OpenAISTTService
    stt = OpenAISTTService(
        api_key="speaches",
        base_url="http://speaches:8000/v1",
        model="Systran/faster-whisper-large-v3-turbo",
        language="en",
    )
    assert "speaches" in stt._client.base_url.host


def test_llm_routes_through_openclaw():
    """LLM must be configured to use OpenClaw gateway, not Claude API directly."""
    from pipecat.services.openai import OpenAILLMService
    llm = OpenAILLMService(
        api_key="test-token",
        base_url="http://openclaw-voicecoach:3000/v1",
        model="openclaw/voicecoach",
        max_tokens=512,
    )
    assert "openclaw-voicecoach" in llm._client.base_url.host
    assert llm.model == "openclaw/voicecoach"


def test_tts_uses_kokoro_not_elevenlabs():
    """TTS must be configured to use Speaches/Kokoro, not ElevenLabs."""
    from pipecat.services.openai import OpenAITTSService
    tts = OpenAITTSService(
        api_key="speaches",
        base_url="http://speaches:8000/v1",
        model="kokoro",
        voice="af_nova",
    )
    assert "speaches" in tts._client.base_url.host
    assert tts.model == "kokoro"


def test_no_external_api_keys_in_pipeline():
    """Verify pipeline module doesn't import Deepgram or ElevenLabs services."""
    import ast
    import inspect
    import pipeline
    source = inspect.getsource(pipeline)
    assert "DeepgramSTTService" not in source, "Deepgram found in pipeline — remove it"
    assert "ElevenLabsTTSService" not in source, "ElevenLabs found in pipeline — remove it"
    assert "AnthropicLLMService" not in source, "Direct Anthropic service found — route through OpenClaw"

---

Step 6: Updated docker-stack.yml

version: "3.8"

networks:
  hobbs-overlay:
    external: true
  voicecoach-internal:
    driver: overlay
    internal: true

volumes:
  speaches-models:
    driver: local
  speechbrain-models:
    driver: local
  openclaw-voicecoach-data:
    driver: local
  livekit-data:
    driver: local
  voicecoach-clips:
    driver: local
  voicecoach-labels:
    driver: local

secrets:
  voicecoach_anthropic_key:
    external: true
  voicecoach_openclaw_token:
    external: true
  voicecoach_livekit_api_key:
    external: true
  voicecoach_livekit_api_secret:
    external: true
  # Matrix secrets (for active labeling — from ACTIVE_LABELING.md)
  voicecoach_matrix_token:
    external: true
  voicecoach_matrix_bot_id:
    external: true
  voicecoach_matrix_jeremy_id:
    external: true
  voicecoach_matrix_homeserver:
    external: true

services:

  # ── LiveKit (unchanged from v1) ────────────────────────────────────────────
  livekit-server:
    image: livekit/livekit-server:latest
    command: --config /etc/livekit/livekit.yaml
    ports:
      - "7880:7880"
      - "7881:7881"
      - "50000-60000:50000-60000/udp"
    volumes:
      - ./livekit/livekit.yaml:/etc/livekit/livekit.yaml:ro
      - livekit-data:/data
    secrets:
      - voicecoach_livekit_api_key
      - voicecoach_livekit_api_secret
    networks:
      - hobbs-overlay
      - voicecoach-internal
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.livekit.rule=Host(`voice.hobbs.farm`)"
        - "traefik.http.routers.livekit.entrypoints=websecure"
        - "traefik.http.routers.livekit.tls.certresolver=cloudflare"
        - "traefik.http.services.livekit.loadbalancer.server.port=7880"

  # ── Speaches: self-hosted Whisper STT + Kokoro TTS ─────────────────────────
  speaches:
    image: fedirz/faster-whisper-server:latest-cpu
    environment:
      - WHISPER__MODEL=large-v3-turbo
      - WHISPER__INFERENCE_DEVICE=cpu
      - WHISPER__COMPUTE_TYPE=int8
      - TTS__MODEL=kokoro
      - TTS__DEFAULT_VOICE=af_nova
      - TTS__LANGUAGE=en-us
    volumes:
      - speaches-models:/root/.cache
    networks:
      - voicecoach-internal
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "3"
          memory: "4G"
        reservations:
          memory: "2G"
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 120s

  # ── SpeechBrain (unchanged from v1) ───────────────────────────────────────
  speechbrain-service:
    image: hobbs.farm/voicecoach/speechbrain:latest
    volumes:
      - speechbrain-models:/app/models
      - voicecoach-clips:/data/clips
    environment:
      - MODEL_DIR=/app/models
    networks:
      - voicecoach-internal
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "2"
          memory: "3G"
        reservations:
          memory: "2G"
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "python", "-c", "import httpx; httpx.get('http://localhost:8001/health').raise_for_status()"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 90s

  # ── OpenClaw VoiceCoach Agent ──────────────────────────────────────────────
  openclaw-voicecoach:
    image: ghcr.io/openclaw/openclaw:latest
    volumes:
      - ./openclaw-agent/openclaw.json:/app/openclaw.json:ro
      - ./openclaw-agent/system_prompt.md:/app/system_prompt.md:ro
      - ./openclaw-agent/skills:/app/skills:ro
      - openclaw-voicecoach-data:/app/data
    secrets:
      - voicecoach_anthropic_key
      - voicecoach_openclaw_token
    environment:
      - OPENCLAW_CONFIG=/app/openclaw.json
      - ANTHROPIC_API_KEY_FILE=/run/secrets/voicecoach_anthropic_key
      - OPENCLAW_TOKEN_FILE=/run/secrets/voicecoach_openclaw_token
    networks:
      - voicecoach-internal
      - hobbs-overlay    # Needs internet for Claude API only
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
      interval: 30s
      timeout: 5s
      retries: 3

  # ── Pipecat Pipeline Agent ─────────────────────────────────────────────────
  pipecat-agent:
    image: hobbs.farm/voicecoach/pipecat:latest
    environment:
      - LIVEKIT_URL=ws://livekit-server:7880
      - SPEACHES_URL=http://speaches:8000
      - SPEECHBRAIN_URL=http://speechbrain-service:8001
      - OPENCLAW_URL=http://openclaw-voicecoach:3000
      - OPENCLAW_AGENT_ID=voicecoach
      - WHISPER_MODEL=Systran/faster-whisper-large-v3-turbo
      - KOKORO_VOICE=af_nova
      - CLIPS_DIR=/data/clips/pending
    secrets:
      - voicecoach_livekit_api_key
      - voicecoach_livekit_api_secret
      - voicecoach_openclaw_token
    volumes:
      - voicecoach-clips:/data/clips
    networks:
      - voicecoach-internal
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 10s

  # ── Web Client (unchanged from v1) ────────────────────────────────────────
  voicecoach-web:
    image: hobbs.farm/voicecoach/web:latest
    networks:
      - hobbs-overlay
      - voicecoach-internal
    deploy:
      replicas: 1
      labels:
        - "traefik.enable=true"
        - "traefik.http.routers.voicecoach.rule=Host(`coach.hobbs.farm`)"
        - "traefik.http.routers.voicecoach.entrypoints=websecure"
        - "traefik.http.routers.voicecoach.tls.certresolver=cloudflare"
        - "traefik.http.services.voicecoach.loadbalancer.server.port=80"
        - "traefik.http.routers.voicecoach.middlewares=voicecoach-auth"
        - "traefik.http.middlewares.voicecoach-auth.basicauth.users=jeremy:$$apr1$$REPLACEME"

---

Step 7: Updated Smoke Test

scripts/smoke-test.sh (v2 additions)

#!/bin/bash
echo "=== VoiceCoach v2 Smoke Tests ==="

# All v1 checks plus:

# 6. Speaches STT health
echo ""
echo "→ Speaches STT health check..."
docker run --rm --network voicecoach_voicecoach-internal \
  curlimages/curl:latest \
  curl -sf http://speaches:8000/health && echo "✅ Speaches responding" || echo "❌ Speaches not responding"

# 7. OpenClaw OpenAI-compatible endpoint
echo ""
echo "→ OpenClaw gateway /v1/models check..."
OPENCLAW_TOKEN=$(docker secret inspect voicecoach_openclaw_token 2>/dev/null | head -1)
docker run --rm --network voicecoach_voicecoach-internal \
  curlimages/curl:latest \
  curl -sf http://openclaw-voicecoach:3000/v1/models \
    -H "Authorization: Bearer dummy-check" \
  && echo "✅ OpenClaw gateway responding" \
  || echo "❌ OpenClaw gateway not responding — check openclaw.json gateway.openaiApi.enabled"

# 8. Confirm no external API calls in pipeline
echo ""
echo "→ Checking pipeline for external service calls..."
if grep -r "deepgram\|elevenlabs\|api.anthropic" pipecat-agent/pipeline.py 2>/dev/null; then
  echo "❌ External service found in pipeline.py"
else
  echo "✅ Pipeline is clean — all services internal or OpenClaw-routed"
fi

echo ""
echo "=== v2 smoke tests complete ==="

---

Step 8: Latency Expectations (v2 vs v1)

Stage	v1 (external)	v2 (self-hosted)	Notes
STT (Deepgram → Speaches)	~200ms	~400-600ms CPU	Faster on GPU
Vocal analysis (SpeechBrain)	~800ms CPU	~800ms CPU	Unchanged
LLM via OpenClaw (→ Claude)	~1200ms	~1300ms	+100ms for OpenClaw hop
TTS (ElevenLabs → Kokoro)	~400ms	~300-500ms CPU	Kokoro is very fast
Total	~2.7s	~3.2s	Within acceptable range

The ~500ms increase is the cost of eliminating all per-minute/per-character fees and routing everything through your infrastructure. For a commute coaching session this is imperceptible.

To recover latency if needed:

• Add GPU to Speaches: image: fedirz/faster-whisper-server:latest-cuda — cuts STT to ~80ms
• Speaches Kokoro TTS on GPU: cuts TTS to ~100ms
• This brings total back under 2s

---

Deploy Order (v2)

# 1. Create secrets (trimmed — no Deepgram/ElevenLabs)
bash scripts/create-secrets.sh

# 2. Build images (speechbrain + pipecat + web only — speaches uses official image)
docker build -t hobbs.farm/voicecoach/speechbrain:latest ./speechbrain-service
docker build -t hobbs.farm/voicecoach/pipecat:latest ./pipecat-agent
docker build -t hobbs.farm/voicecoach/web:latest ./web-client

# 3. Deploy stack
docker stack deploy -c docker-stack.yml voicecoach

# 4. Watch startup — Speaches downloads models on first boot (~2min)
watch docker service ls

# 5. Smoke test
bash scripts/smoke-test.sh

# 6. Run unit tests
cd pipecat-agent && python -m pytest tests/ -v

# 7. Open coach.hobbs.farm — verify all audio is local
#    You should see no calls to api.elevenlabs.io or api.deepgram.com in
#    your router/firewall logs except for claude API calls from openclaw-voicecoach

---

Rollback to v1 (if needed)

# Swap pipeline.py back to v1 version (uses Deepgram + ElevenLabs)
# Requires re-creating the removed secrets:
echo "$DEEPGRAM_KEY"       | docker secret create voicecoach_deepgram_key -
echo "$ELEVENLABS_KEY"     | docker secret create voicecoach_elevenlabs_key -
echo "$ELEVENLABS_VOICE_ID" | docker secret create voicecoach_elevenlabs_voice_id -

# Redeploy
docker stack deploy -c docker-stack-v1.yml voicecoach