feat: initial project scaffold

- Android Auto CarAppService, Session, Screen - Voice pipeline: STT (SpeechManager), TTS (TTSManager), VoiceStateMachine - Matrix REST client (login, send, sync) - no E2EE, raw API - FCM push receiver with response dedup - Network monitor + offline message queue - Phone setup activity stub - Full architecture spec in docs/ARCHITECTURE.md
2026-03-05 22:07:30 +00:00
parent 6c76d18f58
commit 318867b133
23 changed files with 1371 additions and 2 deletions
@@ -0,0 +1,298 @@
+# ClippyAuto 📎🚗 — Final Spec (3 Rounds Complete)
+
+## Vision
+A lightweight Android app purpose-built for voice-first interaction with Clippy via Matrix, optimized for Android Auto commute use.
+
+## Core User Story
+> As Jeremy, I get in my car, Android Auto launches, I tap "Clippy" (or it auto-starts), and I'm in a live voice conversation with my AI Chief of Staff. I talk, Clippy responds in my car speakers within seconds. No typing, no "Hey Google," no Element.
+
+---
+
+## Locked Decisions
+
+| Decision | Resolution | Source |
+|---|---|---|
+| Language | Kotlin | Spirit R1 |
+| Matrix SDK | Raw REST API (Ktor) — no E2EE needed for private homeserver | Spirit R3 |
+| STT | Android SpeechRecognizer (Vosk offline fallback in v2) | Spirit R1 |
+| TTS | Android native TextToSpeech | Spirit R1 |
+| Audio focus | AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK / USAGE_ASSISTANT | Spirit R1 |
+| Auth | Access token in Android Keystore | Spirit R1 |
+| Push delivery | FCM via n8n + Matrix sync dual-path, first wins | Clippy R2 |
+| Music integration | None — Android handles ducking | Margaret R2 |
+| Private mode | Voice command "go quiet" / "speak" | Margaret R2 |
+| Interruption | 200ms TTS fade, activate STT, cancel queued chunks | Clippy R2 |
+| Delegation | Always confirm verbally before sending | Clippy R2 |
+| Audio cue (heard you) | Subtle chime at 2s silence mark, not voice | Clippy R2 |
+| Thinking indicator | Low ambient tone after 3s processing | Clippy R2 |
+| Session resume | < 15 min = resume, > 15 min = new session | Margaret R2, Clippy R2 |
+| Sentence chunking | v1 scope — each sentence as separate Matrix message | Clippy R2 |
+| Dedicated room | #clippy-voice with custom event fields | Margaret R2 |
+| E2EE | Skip for v1 (private homeserver, invite-only room) | Clippy R3 |
+
+---
+
+## Functional Requirements
+
+### FR-1: Voice Input
+- On-device STT via Android SpeechRecognizer
+- Push-to-talk button on Android Auto screen (large, driver-safe)
+- 2-second silence threshold with subtle confirmation chime
+- Cancel-send if Jeremy resumes talking within 1s of chime
+
+### FR-2: Matrix Integration
+- Raw REST API client (Ktor) — login, sync, send
+- Authenticates as Jeremy (dedicated device token)
+- Sends transcribed text to #clippy-voice room
+- Custom event fields: `farm.hobbs.clippy.source`, `farm.hobbs.clippy.session_id`, `farm.hobbs.clippy.signal`
+
+### FR-3: Voice Output
+- Android native TextToSpeech
+- Plays through car audio via USAGE_ASSISTANT audio attributes
+- Queues chunked responses, plays sequentially
+- 200ms fade on interruption (not hard cut)
+
+### FR-4: Android Auto Integration
+- CarAppService with custom Screen template
+- Minimal UI: status indicator (listening/thinking/speaking), talk button
+- Auto-launch option when Android Auto connects (FR-6)
+- Complies with driver distraction guidelines
+
+### FR-5: Session Awareness
+- session_start signal on app open → triggers arrival briefing
+- session_end signal on close/timeout → triggers close summary
+- session_resume for reconnects < 15 min
+
+### FR-6: Auto-Launch
+- Start ClippyAuto automatically when Android Auto connects
+
+### FR-7: Session Context Injection
+- App sends `[ClippyAuto session started — commute mode, voice-first]` to Matrix room
+- Clippy shifts to debrief mode automatically
+
+### FR-8: Debrief Summary
+- On session end, Clippy posts written summary (formatted, markdown OK)
+- Phone UI surfaces as notification: "Debrief summary ready"
+
+### FR-9: Voice Commands
+| Command | Action |
+|---|---|
+| "Clippy, go quiet" | Mute TTS, switch to phone text display |
+| "Clippy, I'm back" / "speak" | Resume voice mode |
+| "Clippy, wrap it up" | Trigger session close + summary |
+| "Clippy, remind me [thing]" | Create reminder in memory |
+| "Clippy, tell [agent] [message]" | Delegation with verbal confirmation |
+
+### FR-10: Silence Detection Tuning
+- 2s pause = processing trigger
+- Subtle confirmation chime at 2s mark
+- If speech resumes within 1s of chime, cancel send and keep listening
+
+### FR-11: Response Length Guardrails
+- App-side: max 250 chars per TTS chunk
+- Split responses > 500 chars into segments with brief pause
+- Clippy enforces 2-sentence max per message in voice mode
+
+---
+
+## Non-Functional Requirements
+
+| ID | Requirement | Target |
+|---|---|---|
+| NFR-1 | Time to first audio | < 5 seconds (with FCM + chunking) |
+| NFR-2 | Offline grace | Queue messages, notify verbally, resume on reconnect |
+| NFR-3 | Battery | Minimal drain; only active in foreground |
+| NFR-4 | Privacy | All voice processing on-device; only text hits network |
+| NFR-5 | Eyes-free | Zero required visual interactions after launch |
+
+---
+
+## Architecture
+
+```
+┌─────────────────────────────────────┐
+│         Android Auto Screen         │
+│   [ 🎙️ Talk to Clippy ] [Status]   │
+└──────────────┬──────────────────────┘
+               │
+┌──────────────▼──────────────────────┐
+│         ClippyAuto App              │
+│  ┌───────────┐  ┌────────────────┐  │
+│  │ STT Engine│  │ TTS Engine     │  │
+│  │ (on-device)│  │ (on-device)   │  │
+│  └─────┬─────┘  └───────▲────────┘  │
+│        │ text           │ text      │
+│  ┌─────▼────────────────┴────────┐  │
+│  │  Matrix REST Client (Ktor)    │  │
+│  │  Room: #clippy-voice          │  │
+│  │  + FCM receiver (fast path)   │  │
+│  └─────┬─────────────────▲───────┘  │
+└────────┼─────────────────┼──────────┘
+         │                 │
+    ┌────▼─────────────────┴────┐
+    │   mess.hobbs.farm:8008    │
+    │   Matrix Synapse          │
+    │          │                │
+    │   Push gateway → n8n     │
+    │          │                │
+    │          ▼                │
+    │   FCM data message       │
+    └────┬─────────────────▲────┘
+         │                 │
+    ┌────▼─────────────────┴────┐
+    │   Clippy (OpenClaw Agent) │
+    │   Claude                  │
+    └───────────────────────────┘
+```
+
+### Push Path (Fast)
+```
+Clippy sends message → Synapse push gateway → n8n webhook
+  → n8n filters (sender == @clippy) → FCM data message → app → TTS
+Latency: ~400-700ms from Clippy message to TTS start
+```
+
+### Sync Path (Fallback)
+```
+Clippy sends message → Matrix sync long-poll → app → TTS
+Latency: ~1-2s (depends on poll timing)
+```
+
+Both paths active; first to deliver wins; dedup via hash.
+
+---
+
+## Latency Budget
+
+| Step | Target | Notes |
+|---|---|---|
+| STT | 800ms | On-device |
+| Matrix send | 400ms | Cellular |
+| Claude to first sentence | 1.5-2.5s | Sentence chunking |
+| FCM delivery | 300ms | High priority data message |
+| TTS start | 300ms | Pre-initialized |
+| **Total to first audio** | **3.3-4.3s** | 🎯 Under 5 seconds |
+
+---
+
+## Voice State Machine
+
+```
+         ┌──────────┐
+         │   IDLE   │◄──────────────────────┐
+         └────┬─────┘                       │
+              │ tap button                  │ TTS complete
+         ┌────▼─────┐                  ┌────┴─────┐
+         │LISTENING │                  │ SPEAKING  │
+         └────┬─────┘                  └────▲─────┘
+              │ 2s silence + chime          │ response received
+         ┌────▼──────┐                 ┌────┴──────┐
+         │PROCESSING │────────────────►│ RECEIVING │
+         └───────────┘  sent to Matrix └───────────┘
+```
+
+---
+
+## Connectivity Handling
+
+| Scenario | Behavior |
+|---|---|
+| Tunnel / dead zone | Queue outgoing, TTS: "Signal dropped, I'll catch up." |
+| Clippy slow (>10s) | TTS: "Still thinking on that one..." |
+| Bluetooth disconnect | Pause session, resume on reconnect |
+| Resume < 15 min | session_resume, "Welcome back" |
+| Resume > 15 min | session_start, fresh arrival briefing |
+
+---
+
+## Clippy Voice Mode Prompt (appended to SOUL.md during voice sessions)
+
+```
+VOICE MODE RULES:
+1. Max 2 sentences per message. Hard limit.
+2. Send each sentence as a separate Matrix message (enables chunked TTS).
+3. No markdown. No bullet points. No formatting. Plain speech only.
+4. No emoji in voice responses.
+5. Numbers: say "three" not "3". Say "two thirty" not "2:30 PM".
+6. Names: full name first mention, first name after.
+7. Don't say "let me think about that" — just think and respond.
+8. Lists conversationally: "You've got three things: X, Y, and Z."
+
+ARRIVAL BRIEFING (on session_start):
+  Message 1: "Hey Jeremy. [Most urgent/time-sensitive thing]."
+  Message 2: "[One open loop from last session or overnight change]."
+  Then: silence.
+
+SESSION CLOSE (on "wrap it up" or session_end):
+  Message 1: "Alright, here's what I've got. [Action items conversationally]."
+  Message 2: "[Anything unresolved carrying to next session]."
+  Then: post written markdown summary to room.
+
+SESSION RESUME (on session_resume):
+  Message 1: "Welcome back. [Where we left off, one sentence]."
+  Then: silence.
+```
+
+---
+
+## Security
+
+| Concern | Mitigation |
+|---|---|
+| Token storage | Android Keystore (hardware-backed) |
+| Network | TLS to homeserver |
+| FCM payload | Data message only, no sensitive text in notification |
+| Voice data | Never stored; STT real-time, audio buffer discarded |
+| Phone screen | Locked during Android Auto; transcript after session |
+
+---
+
+## v1 MVP Scope
+
+**Ships:**
+- Push-to-talk Android Auto interface
+- On-device STT → Matrix REST → Clippy → FCM → TTS
+- Sentence-level chunked responses (sub-5s first audio)
+- Arrival briefing, session close summary
+- Voice commands (quiet/speak/wrap up/remind/delegate)
+- Private/passenger mode
+- Connectivity handling
+- Dedicated #clippy-voice Matrix room
+- Minimal phone UI (setup + session active)
+
+**v2 Backlog:**
+- Wake word ("Hey Clippy") via Porcupine
+- Google Calendar integration
+- Direct SSE streaming endpoint
+- Offline STT (Vosk fallback)
+- Full phone transcript review UI
+- Agent overnight digest system
+
+---
+
+## Build Estimate
+
+| Component | Days |
+|---|---|
+| Project setup + Android Auto scaffold | 2 |
+| Matrix REST client + auth | 1.5 |
+| Voice pipeline (STT + TTS + state machine) | 3 |
+| FCM push path (app side) | 1 |
+| n8n workflow (server side) | 0.5 |
+| Clippy voice mode prompt tuning | 1 |
+| Android Auto UI | 2 |
+| Phone UI (minimal) | 1 |
+| Testing + polish | 3 |
+| **Total** | **~15 days** |
+
+---
+
+## Open Questions for Jeremy
+
+1. **Build approach** — Who builds this? Spirit scaffolds? Jeremy builds with our spec? Fork existing project?
+2. **Wake word** — "Hey Clippy" for v2, or push-to-talk fine indefinitely?
+3. **Which Google Calendar(s)** for v2?
+4. **Agent overnight reports** — Want Margaret/Spirit posting daily summaries for Clippy to pull from?
+5. **Homeserver external access** — Is mess.hobbs.farm already reachable via Traefik externally?
+6. **Firebase project** — Existing one or create new?
+7. **Testing** — Wired Android Auto first, or straight to wireless?