# ClippyAuto 📎🚗 — Final Spec (3 Rounds Complete)

## Vision
A lightweight Android app purpose-built for voice-first interaction with Clippy via Matrix, optimized for Android Auto commute use.

## Core User Story
> As Jeremy, I get in my car, Android Auto launches, I tap "Clippy" (or it auto-starts), and I'm in a live voice conversation with my AI Chief of Staff. I talk, Clippy responds in my car speakers within seconds. No typing, no "Hey Google," no Element.

---

## Locked Decisions

| Decision | Resolution | Source |
|---|---|---|
| Language | Kotlin | Spirit R1 |
| Matrix SDK | Raw REST API (Ktor) — no E2EE needed for private homeserver | Spirit R3 |
| STT | Android SpeechRecognizer (Vosk offline fallback in v2) | Spirit R1 |
| TTS | Android native TextToSpeech | Spirit R1 |
| Audio focus | AUDIOFOCUS_GAIN_TRANSIENT_MAY_DUCK / USAGE_ASSISTANT | Spirit R1 |
| Auth | Access token in Android Keystore | Spirit R1 |
| Push delivery | FCM via n8n + Matrix sync dual-path, first wins | Clippy R2 |
| Music integration | None — Android handles ducking | Margaret R2 |
| Private mode | Voice command "go quiet" / "speak" | Margaret R2 |
| Interruption | 200ms TTS fade, activate STT, cancel queued chunks | Clippy R2 |
| Delegation | Always confirm verbally before sending | Clippy R2 |
| Audio cue (heard you) | Subtle chime at 2s silence mark, not voice | Clippy R2 |
| Thinking indicator | Low ambient tone after 3s processing | Clippy R2 |
| Session resume | < 15 min = resume, > 15 min = new session | Margaret R2, Clippy R2 |
| Sentence chunking | v1 scope — each sentence as separate Matrix message | Clippy R2 |
| Dedicated room | #clippy-voice with custom event fields | Margaret R2 |
| E2EE | Skip for v1 (private homeserver, invite-only room) | Clippy R3 |

---

## Functional Requirements

### FR-1: Voice Input
- On-device STT via Android SpeechRecognizer
- Push-to-talk button on Android Auto screen (large, driver-safe)
- 2-second silence threshold with subtle confirmation chime
- Cancel-send if Jeremy resumes talking within 1s of chime

### FR-2: Matrix Integration
- Raw REST API client (Ktor) — login, sync, send
- Authenticates as Jeremy (dedicated device token)
- Sends transcribed text to #clippy-voice room
- Custom event fields: `farm.hobbs.clippy.source`, `farm.hobbs.clippy.session_id`, `farm.hobbs.clippy.signal`

### FR-3: Voice Output
- Android native TextToSpeech
- Plays through car audio via USAGE_ASSISTANT audio attributes
- Queues chunked responses, plays sequentially
- 200ms fade on interruption (not hard cut)

### FR-4: Android Auto Integration
- CarAppService with custom Screen template
- Minimal UI: status indicator (listening/thinking/speaking), talk button
- Auto-launch option when Android Auto connects (FR-6)
- Complies with driver distraction guidelines

### FR-5: Session Awareness
- session_start signal on app open → triggers arrival briefing
- session_end signal on close/timeout → triggers close summary
- session_resume for reconnects < 15 min

### FR-6: Auto-Launch
- Start ClippyAuto automatically when Android Auto connects

### FR-7: Session Context Injection
- App sends `[ClippyAuto session started — commute mode, voice-first]` to Matrix room
- Clippy shifts to debrief mode automatically

### FR-8: Debrief Summary
- On session end, Clippy posts written summary (formatted, markdown OK)
- Phone UI surfaces as notification: "Debrief summary ready"

### FR-9: Voice Commands
| Command | Action |
|---|---|
| "Clippy, go quiet" | Mute TTS, switch to phone text display |
| "Clippy, I'm back" / "speak" | Resume voice mode |
| "Clippy, wrap it up" | Trigger session close + summary |
| "Clippy, remind me [thing]" | Create reminder in memory |
| "Clippy, tell [agent] [message]" | Delegation with verbal confirmation |

### FR-10: Silence Detection Tuning
- 2s pause = processing trigger
- Subtle confirmation chime at 2s mark
- If speech resumes within 1s of chime, cancel send and keep listening

### FR-11: Response Length Guardrails
- App-side: max 250 chars per TTS chunk
- Split responses > 500 chars into segments with brief pause
- Clippy enforces 2-sentence max per message in voice mode

---

## Non-Functional Requirements

| ID | Requirement | Target |
|---|---|---|
| NFR-1 | Time to first audio | < 5 seconds (with FCM + chunking) |
| NFR-2 | Offline grace | Queue messages, notify verbally, resume on reconnect |
| NFR-3 | Battery | Minimal drain; only active in foreground |
| NFR-4 | Privacy | All voice processing on-device; only text hits network |
| NFR-5 | Eyes-free | Zero required visual interactions after launch |

---

## Architecture

```
┌─────────────────────────────────────┐
│         Android Auto Screen         │
│   [ 🎙️ Talk to Clippy ] [Status]   │
└──────────────┬──────────────────────┘
               │
┌──────────────▼──────────────────────┐
│         ClippyAuto App              │
│  ┌───────────┐  ┌────────────────┐  │
│  │ STT Engine│  │ TTS Engine     │  │
│  │ (on-device)│  │ (on-device)   │  │
│  └─────┬─────┘  └───────▲────────┘  │
│        │ text           │ text      │
│  ┌─────▼────────────────┴────────┐  │
│  │  Matrix REST Client (Ktor)    │  │
│  │  Room: #clippy-voice          │  │
│  │  + FCM receiver (fast path)   │  │
│  └─────┬─────────────────▲───────┘  │
└────────┼─────────────────┼──────────┘
         │                 │
    ┌────▼─────────────────┴────┐
    │   mess.hobbs.farm:8008    │
    │   Matrix Synapse          │
    │          │                │
    │   Push gateway → n8n     │
    │          │                │
    │          ▼                │
    │   FCM data message       │
    └────┬─────────────────▲────┘
         │                 │
    ┌────▼─────────────────┴────┐
    │   Clippy (OpenClaw Agent) │
    │   Claude                  │
    └───────────────────────────┘
```

### Push Path (Fast)
```
Clippy sends message → Synapse push gateway → n8n webhook
  → n8n filters (sender == @clippy) → FCM data message → app → TTS
Latency: ~400-700ms from Clippy message to TTS start
```

### Sync Path (Fallback)
```
Clippy sends message → Matrix sync long-poll → app → TTS
Latency: ~1-2s (depends on poll timing)
```

Both paths active; first to deliver wins; dedup via hash.

---

## Latency Budget

| Step | Target | Notes |
|---|---|---|
| STT | 800ms | On-device |
| Matrix send | 400ms | Cellular |
| Claude to first sentence | 1.5-2.5s | Sentence chunking |
| FCM delivery | 300ms | High priority data message |
| TTS start | 300ms | Pre-initialized |
| **Total to first audio** | **3.3-4.3s** | 🎯 Under 5 seconds |

---

## Voice State Machine

```
         ┌──────────┐
         │   IDLE   │◄──────────────────────┐
         └────┬─────┘                       │
              │ tap button                  │ TTS complete
         ┌────▼─────┐                  ┌────┴─────┐
         │LISTENING │                  │ SPEAKING  │
         └────┬─────┘                  └────▲─────┘
              │ 2s silence + chime          │ response received
         ┌────▼──────┐                 ┌────┴──────┐
         │PROCESSING │────────────────►│ RECEIVING │
         └───────────┘  sent to Matrix └───────────┘
```

---

## Connectivity Handling

| Scenario | Behavior |
|---|---|
| Tunnel / dead zone | Queue outgoing, TTS: "Signal dropped, I'll catch up." |
| Clippy slow (>10s) | TTS: "Still thinking on that one..." |
| Bluetooth disconnect | Pause session, resume on reconnect |
| Resume < 15 min | session_resume, "Welcome back" |
| Resume > 15 min | session_start, fresh arrival briefing |

---

## Clippy Voice Mode Prompt (appended to SOUL.md during voice sessions)

```
VOICE MODE RULES:
1. Max 2 sentences per message. Hard limit.
2. Send each sentence as a separate Matrix message (enables chunked TTS).
3. No markdown. No bullet points. No formatting. Plain speech only.
4. No emoji in voice responses.
5. Numbers: say "three" not "3". Say "two thirty" not "2:30 PM".
6. Names: full name first mention, first name after.
7. Don't say "let me think about that" — just think and respond.
8. Lists conversationally: "You've got three things: X, Y, and Z."

ARRIVAL BRIEFING (on session_start):
  Message 1: "Hey Jeremy. [Most urgent/time-sensitive thing]."
  Message 2: "[One open loop from last session or overnight change]."
  Then: silence.

SESSION CLOSE (on "wrap it up" or session_end):
  Message 1: "Alright, here's what I've got. [Action items conversationally]."
  Message 2: "[Anything unresolved carrying to next session]."
  Then: post written markdown summary to room.

SESSION RESUME (on session_resume):
  Message 1: "Welcome back. [Where we left off, one sentence]."
  Then: silence.
```

---

## Security

| Concern | Mitigation |
|---|---|
| Token storage | Android Keystore (hardware-backed) |
| Network | TLS to homeserver |
| FCM payload | Data message only, no sensitive text in notification |
| Voice data | Never stored; STT real-time, audio buffer discarded |
| Phone screen | Locked during Android Auto; transcript after session |

---

## v1 MVP Scope

**Ships:**
- Push-to-talk Android Auto interface
- On-device STT → Matrix REST → Clippy → FCM → TTS
- Sentence-level chunked responses (sub-5s first audio)
- Arrival briefing, session close summary
- Voice commands (quiet/speak/wrap up/remind/delegate)
- Private/passenger mode
- Connectivity handling
- Dedicated #clippy-voice Matrix room
- Minimal phone UI (setup + session active)

**v2 Backlog:**
- Wake word ("Hey Clippy") via Porcupine
- Google Calendar integration
- Direct SSE streaming endpoint
- Offline STT (Vosk fallback)
- Full phone transcript review UI
- Agent overnight digest system

---

## Build Estimate

| Component | Days |
|---|---|
| Project setup + Android Auto scaffold | 2 |
| Matrix REST client + auth | 1.5 |
| Voice pipeline (STT + TTS + state machine) | 3 |
| FCM push path (app side) | 1 |
| n8n workflow (server side) | 0.5 |
| Clippy voice mode prompt tuning | 1 |
| Android Auto UI | 2 |
| Phone UI (minimal) | 1 |
| Testing + polish | 3 |
| **Total** | **~15 days** |

---

## Open Questions for Jeremy

1. **Build approach** — Who builds this? Spirit scaffolds? Jeremy builds with our spec? Fork existing project?
2. **Wake word** — "Hey Clippy" for v2, or push-to-talk fine indefinitely?
3. **Which Google Calendar(s)** for v2?
4. **Agent overnight reports** — Want Margaret/Spirit posting daily summaries for Clippy to pull from?
5. **Homeserver external access** — Is mess.hobbs.farm already reachable via Traefik externally?
6. **Firebase project** — Existing one or create new?
7. **Testing** — Wired Android Auto first, or straight to wireless?