clippy-auto/docs/ARCHITECTURE.md

ClippyAuto 📎🚗 — Final Spec (3 Rounds Complete)

Vision

A lightweight Android app purpose-built for voice-first interaction with Clippy via Matrix, optimized for Android Auto commute use.

Core User Story

As Jeremy, I get in my car, Android Auto launches, I tap "Clippy" (or it auto-starts), and I'm in a live voice conversation with my AI Chief of Staff. I talk, Clippy responds in my car speakers within seconds. No typing, no "Hey Google," no Element.

---

Locked Decisions

Decision	Resolution	Source
Language	Kotlin	Spirit R1
Matrix SDK	Raw REST API (Ktor) — no E2EE needed for private homeserver	Spirit R3
STT	Android SpeechRecognizer (Vosk offline fallback in v2)	Spirit R1
TTS	Android native TextToSpeech	Spirit R1
Audio focus	AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK / USAGE_ASSISTANT	Spirit R1
Auth	Access token in Android Keystore	Spirit R1
Push delivery	FCM via n8n + Matrix sync dual-path, first wins	Clippy R2
Music integration	None — Android handles ducking	Margaret R2
Private mode	Voice command "go quiet" / "speak"	Margaret R2
Interruption	200ms TTS fade, activate STT, cancel queued chunks	Clippy R2
Delegation	Always confirm verbally before sending	Clippy R2
Audio cue (heard you)	Subtle chime at 2s silence mark, not voice	Clippy R2
Thinking indicator	Low ambient tone after 3s processing	Clippy R2
Session resume	< 15 min = resume, > 15 min = new session	Margaret R2, Clippy R2
Sentence chunking	v1 scope — each sentence as separate Matrix message	Clippy R2
Dedicated room	#clippy-voice with custom event fields	Margaret R2
E2EE	Skip for v1 (private homeserver, invite-only room)	Clippy R3

---

Functional Requirements

FR-1: Voice Input

• On-device STT via Android SpeechRecognizer
• Push-to-talk button on Android Auto screen (large, driver-safe)
• 2-second silence threshold with subtle confirmation chime
• Cancel-send if Jeremy resumes talking within 1s of chime

FR-2: Matrix Integration

• Raw REST API client (Ktor) — login, sync, send
• Authenticates as Jeremy (dedicated device token)
• Sends transcribed text to #clippy-voice room
• Custom event fields: farm.hobbs.clippy.source, farm.hobbs.clippy.session_id, farm.hobbs.clippy.signal

FR-3: Voice Output

• Android native TextToSpeech
• Plays through car audio via USAGE_ASSISTANT audio attributes
• Queues chunked responses, plays sequentially
• 200ms fade on interruption (not hard cut)

FR-4: Android Auto Integration

• CarAppService with custom Screen template
• Minimal UI: status indicator (listening/thinking/speaking), talk button
• Auto-launch option when Android Auto connects (FR-6)
• Complies with driver distraction guidelines

FR-5: Session Awareness

• session_start signal on app open → triggers arrival briefing
• session_end signal on close/timeout → triggers close summary
• session_resume for reconnects < 15 min

FR-6: Auto-Launch

• Start ClippyAuto automatically when Android Auto connects

FR-7: Session Context Injection

• App sends [ClippyAuto session started — commute mode, voice-first] to Matrix room
• Clippy shifts to debrief mode automatically

FR-8: Debrief Summary

• On session end, Clippy posts written summary (formatted, markdown OK)
• Phone UI surfaces as notification: "Debrief summary ready"

FR-9: Voice Commands

Command	Action
"Clippy, go quiet"	Mute TTS, switch to phone text display
"Clippy, I'm back" / "speak"	Resume voice mode
"Clippy, wrap it up"	Trigger session close + summary
"Clippy, remind me [thing]"	Create reminder in memory
"Clippy, tell [agent] [message]"	Delegation with verbal confirmation

FR-10: Silence Detection Tuning

• 2s pause = processing trigger
• Subtle confirmation chime at 2s mark
• If speech resumes within 1s of chime, cancel send and keep listening

FR-11: Response Length Guardrails

• App-side: max 250 chars per TTS chunk
• Split responses > 500 chars into segments with brief pause
• Clippy enforces 2-sentence max per message in voice mode

---

Non-Functional Requirements

ID	Requirement	Target
NFR-1	Time to first audio	< 5 seconds (with FCM + chunking)
NFR-2	Offline grace	Queue messages, notify verbally, resume on reconnect
NFR-3	Battery	Minimal drain; only active in foreground
NFR-4	Privacy	All voice processing on-device; only text hits network
NFR-5	Eyes-free	Zero required visual interactions after launch

---

Architecture

┌─────────────────────────────────────┐
│         Android Auto Screen         │
│   [ 🎙️ Talk to Clippy ] [Status]   │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│         ClippyAuto App              │
│  ┌───────────┐  ┌────────────────┐  │
│  │ STT Engine│  │ TTS Engine     │  │
│  │ (on-device)│  │ (on-device)   │  │
│  └─────┬─────┘  └───────▲────────┘  │
│        │ text           │ text      │
│  ┌─────▼────────────────┴────────┐  │
│  │  Matrix REST Client (Ktor)    │  │
│  │  Room: #clippy-voice          │  │
│  │  + FCM receiver (fast path)   │  │
│  └─────┬─────────────────▲───────┘  │
└────────┼─────────────────┼──────────┘
         │                 │
    ┌────▼─────────────────┴────┐
    │   mess.hobbs.farm:8008    │
    │   Matrix Synapse          │
    │          │                │
    │   Push gateway → n8n     │
    │          │                │
    │          ▼                │
    │   FCM data message       │
    └────┬─────────────────▲────┘
         │                 │
    ┌────▼─────────────────┴────┐
    │   Clippy (OpenClaw Agent) │
    │   Claude                  │
    └───────────────────────────┘

Push Path (Fast)

Clippy sends message → Synapse push gateway → n8n webhook
  → n8n filters (sender == @clippy) → FCM data message → app → TTS
Latency: ~400-700ms from Clippy message to TTS start

Sync Path (Fallback)

Clippy sends message → Matrix sync long-poll → app → TTS
Latency: ~1-2s (depends on poll timing)

Both paths active; first to deliver wins; dedup via hash.

---

Latency Budget

Step	Target	Notes
STT	800ms	On-device
Matrix send	400ms	Cellular
Claude to first sentence	1.5-2.5s	Sentence chunking
FCM delivery	300ms	High priority data message
TTS start	300ms	Pre-initialized
Total to first audio	3.3-4.3s	🎯 Under 5 seconds

---

Voice State Machine

         ┌──────────┐
         │   IDLE   │◄──────────────────────┐
         └────┬─────┘                       │
              │ tap button                  │ TTS complete
         ┌────▼─────┐                  ┌────┴─────┐
         │LISTENING │                  │ SPEAKING  │
         └────┬─────┘                  └────▲─────┘
              │ 2s silence + chime          │ response received
         ┌────▼──────┐                 ┌────┴──────┐
         │PROCESSING │────────────────►│ RECEIVING │
         └───────────┘  sent to Matrix └───────────┘

---

Connectivity Handling

Scenario	Behavior
Tunnel / dead zone	Queue outgoing, TTS: "Signal dropped, I'll catch up."
Clippy slow (>10s)	TTS: "Still thinking on that one..."
Bluetooth disconnect	Pause session, resume on reconnect
Resume < 15 min	session_resume, "Welcome back"
Resume > 15 min	session_start, fresh arrival briefing

---

Clippy Voice Mode Prompt (appended to SOUL.md during voice sessions)

VOICE MODE RULES:
1. Max 2 sentences per message. Hard limit.
2. Send each sentence as a separate Matrix message (enables chunked TTS).
3. No markdown. No bullet points. No formatting. Plain speech only.
4. No emoji in voice responses.
5. Numbers: say "three" not "3". Say "two thirty" not "2:30 PM".
6. Names: full name first mention, first name after.
7. Don't say "let me think about that" — just think and respond.
8. Lists conversationally: "You've got three things: X, Y, and Z."

ARRIVAL BRIEFING (on session_start):
  Message 1: "Hey Jeremy. [Most urgent/time-sensitive thing]."
  Message 2: "[One open loop from last session or overnight change]."
  Then: silence.

SESSION CLOSE (on "wrap it up" or session_end):
  Message 1: "Alright, here's what I've got. [Action items conversationally]."
  Message 2: "[Anything unresolved carrying to next session]."
  Then: post written markdown summary to room.

SESSION RESUME (on session_resume):
  Message 1: "Welcome back. [Where we left off, one sentence]."
  Then: silence.

---

Security

Concern	Mitigation
Token storage	Android Keystore (hardware-backed)
Network	TLS to homeserver
FCM payload	Data message only, no sensitive text in notification
Voice data	Never stored; STT real-time, audio buffer discarded
Phone screen	Locked during Android Auto; transcript after session

---

v1 MVP Scope

Ships:

• Push-to-talk Android Auto interface
• On-device STT → Matrix REST → Clippy → FCM → TTS
• Sentence-level chunked responses (sub-5s first audio)
• Arrival briefing, session close summary
• Voice commands (quiet/speak/wrap up/remind/delegate)
• Private/passenger mode
• Connectivity handling
• Dedicated #clippy-voice Matrix room
• Minimal phone UI (setup + session active)

v2 Backlog:

• Wake word ("Hey Clippy") via Porcupine
• Google Calendar integration
• Direct SSE streaming endpoint
• Offline STT (Vosk fallback)
• Full phone transcript review UI
• Agent overnight digest system

---

Build Estimate

Component	Days
Project setup + Android Auto scaffold	2
Matrix REST client + auth	1.5
Voice pipeline (STT + TTS + state machine)	3
FCM push path (app side)	1
n8n workflow (server side)	0.5
Clippy voice mode prompt tuning	1
Android Auto UI	2
Phone UI (minimal)	1
Testing + polish	3
Total	~15 days

---

Open Questions for Jeremy

1. Build approach — Who builds this? Spirit scaffolds? Jeremy builds with our spec? Fork existing project?
2. Wake word — "Hey Clippy" for v2, or push-to-talk fine indefinitely?
3. Which Google Calendar(s) for v2?
4. Agent overnight reports — Want Margaret/Spirit posting daily summaries for Clippy to pull from?
5. Homeserver external access — Is mess.hobbs.farm already reachable via Traefik externally?
6. Firebase project — Existing one or create new?
7. Testing — Wired Android Auto first, or straight to wireless?