hobbs-iac/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository Overview

This repository manages a homelab infrastructure with self-hosted services running across 3 nodes using standalone Docker Compose on all nodes. (Docker Swarm was fully dismantled on 2026-04-01.)

Cluster Architecture

Nodes:

• hf-pdocker-01 (192.168.86.192) - Core infrastructure + Frigate NVR - RESTORED (2026-03-08)
• hf-pdocker-02 (192.168.86.100) - Dev tools, utilities
• bart (192.168.86.167) - Media stack, OpenVAS

Synology NAS (192.168.86.30) - Centralized NFS storage for all service data

Access: Username glyptic, password &Rip8a115 (has sudo access on all nodes)

Service Distribution

hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
├── [COMPOSE] clip-service (:5200) -- CLIP embeddings (ViT-L-14, CPU)
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] ESPHome (:6052)
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Synapse/Matrix (:8008) + synapse-db (postgres) + wellknown (:8010)
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
├── [COMPOSE] AdGuard Home (:3000, :53)
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (daily 4am)
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)

bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr)
├── [COMPOSE] Onyx (:3005) -- redeployed 2026-05-20, all services :latest, Watchtower disabled on all 10
├── [COMPOSE] OpenVAS (vulnerability scanner)
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Watchtower

hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
├── [COMPOSE] Unifi + unifi-db (MongoDB)
├── [COMPOSE] Gitea + gitea-db (PostgreSQL)
├── [COMPOSE] Semaphore + semaphore-db (MySQL)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
├── [COMPOSE] LifeOS (:8095) -- personal life/goals/habits tracker (PWA). Local mode: Flask+SQLite, no Supabase. LAN-only (no auth in backend). AI companion -> Ollama on .150.
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Watchtower

Working with This Repository

Environment Context

You are working in a WSL 2 environment on Windows. Docker is NOT directly available in WSL - all Docker operations must be performed by connecting to the remote nodes via SSH.

Critical: Docker is not available locally in WSL. All Docker operations must be performed by connecting to the remote nodes via SSH.

Storage Architecture

All service data lives on the Synology NAS via NFS, mounted at /mnt/docker-data on all 3 nodes.

/mnt/docker-data/ (NAS: 192.168.86.30:/volume1/Docker)
├── configs/          # Application configs
├── databases/        # Database data directories
├── media/            # Frigate recordings
├── traefik/          # Let's Encrypt certs
└── compose/          # Standalone compose files per node
    ├── hf-pdocker-01/{frigate,frigate-notify,clip-service,matrix,smarthome,guacamole,ai-agents,traefik,adguard}/
    ├── hf-pdocker-02/{unifi,gitea,semaphore,wikijs,vaultwarden,actual-budget,mcpo,trivy,watchtower}/
    └── bart/{media,openvas,watchtower}/

NFS mount (all nodes): 192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3,nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s 0 0 Note (hf-pdocker-01): Both NFS mounts hardened with nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s on 2026-05-20 to prevent boot-time race (network not up when NFS tried to mount, cascading ~10 container failures).

NFS + Databases: Database containers on NFS require user: "1024:100" to avoid chown failures from NFS root_squash. UID 1024 is the NAS anonymous user.

Repository File Structure

stacks/              # LEGACY Swarm stack definitions (Swarm dismantled 2026-04-01, kept for reference)

compose/             # Standalone Compose definitions (deployed per-node)
├── hf-pdocker-01/   # Core infrastructure + Frigate
├── hf-pdocker-02/   # Dev tools & utilities
│   ├── {service}/docker-compose.yml
│   └── {service}/.env
└── bart/            # Media stack & OpenVAS

Deployment Workflow

All services (compose/):

1. Edit compose file in compose/{node}/{service}/docker-compose.yml
2. SCP to NAS path or node, then: cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d

Service Architecture Patterns

Network Patterns:

• Compose services: Bridge networks for service isolation
• Public services (Traefik): Traefik labels + dynamic config files in /mnt/docker-data/traefik/dynamic/

Database Patterns:

• PostgreSQL: postgres:15-alpine or postgres:16-alpine
• MySQL: mysql:8.0 (Guacamole, Semaphore)
• MongoDB: mongo:7.0 (Unifi)
• Always add com.centurylinklabs.watchtower.enable=false label
• Always add user: "1024:100" for NFS-backed database volumes

Volume Patterns:

• Application data: /mnt/docker-data/configs/{service-name}/
• Database data: /mnt/docker-data/databases/{service-name}-db/
• Traefik certs: /mnt/docker-data/traefik/letsencrypt/

Key Services & Access

Public Services (via Traefik on hf-pdocker-01)

• Matrix: mess.hobbs.farm
• Guacamole: solo.hobbs.farm
• Home Assistant: yoda.hobbs.farm
• Vaultwarden: vault.hobbs.farm
• Gitea: git.hobbs.farm
• FRIDAY Voice: coach.hobbs.farm (replaced VoiceCoach 2026-07-05; runs on .150 :8320)
• Rongorongo: rongorongo.hobbs.farm
• LinkedStorm: linkedstorm.hobbs.farm

Internal Services by Node

hf-pdocker-01 (192.168.86.192):

• Frigate: :5000
• Frigate Alerts (v2): :5199 (web UI + REST API)
• clip-service: :5200 (CLIP embeddings, ViT-L-14, CPU)
• Home Assistant: :8123
• ESPHome: :6052
• Mosquitto: :1883
• Matrix/Synapse: :8008
• Well-known (Matrix federation): :8010
• Guacamole: :8199
• AdGuard Home: :3000 (DNS: :53)
• Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
• Grafana: :3001
• Prometheus: :9095
• Alertmanager: :9093
• InfluxDB: :8086
• Blackbox Exporter: :9115
• matrix-alertmanager-receiver: :9094
• AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0)
• Farm Manager: :8888 (dashboard + API)
• Farm Agent: :8889
• Rongorongo: :8181 (rongorongo-web nginx; reverse-proxies /api/* to rongorongo-api Flask backend on the internal rongorongo bridge net). State + images at /mnt/docker-data/configs/rongorongo/data/. Public URL gated by Traefik basicAuth (creds in Vaultwarden).
• LinkedStorm: :8300 (pipeline/VAD+STT+speaker-ID), :8310 (API/dashboard), cron (no port). Public: linkedstorm.hobbs.farm -> :8310. Source: /opt/linkedstorm (rsync from WSL). Data: /mnt/docker-data/configs/linkedstorm/. STT via Speaches GPU machine :8000, LLM via Ollama :11434 (qwen2.5:14b).
• VoiceCoach -- fully decommissioned 2026-07-05, replaced by FRIDAY Voice on .150 (see GPU Machine section + memory friday-voice)

hf-pdocker-02 (192.168.86.100):

• Farm Agent: :8889
• Unifi: :8443
• Gitea: :3003 (web), :2222 (SSH)
• Semaphore: :3004
• Wiki.js: :3002
• Vaultwarden: :26226
• Actual Budget: :5006
• MCPO: :8001
• Trivy: :4954
• LifeOS: :8095 (host) -> :8080 (container). LAN-only, deliberately NOT behind Traefik: the Flask backend is single-user with NO auth. Image lifeos:local built on-node from /opt/lifeos (upstream https://github.com/lifeos-app/lifeos, pinned 3009bf6). SQLite at /var/lib/lifeos/data.db on LOCAL disk (NFS breaks SQLite WAL locking). Compose: compose/hf-pdocker-02/lifeos/. Note: host 8080 is taken by unifi-os-server.

bart (192.168.86.167):

• Farm Agent: :8889
• Jellyfin: :8096
• Sonarr: :8989
• Radarr: :7878
• SABnzbd: :8082
• NZBHydra2: :5076
• Jellyseerr: :5055
• OpenVAS: :9392

GPU Machine (192.168.86.150, RTX 3090):

• NOTE (2026-07-05): .150 is actually the user's Windows workstation (runs the WSL/Claude Code session), not a headless Linux server. It runs Docker Desktop (WSL2 backend, RTX 3090 GPU passthrough working) alongside UniFi OS Server (Podman). Docker Desktop panics if UniFi/Podman claims \\.\pipe\docker_engine first — start Docker Desktop before UniFi OS Server. See memory docker-desktop-unifi-podman-conflict.
• FRIDAY Voice replaced VoiceCoach (2026-07-05). VoiceCoach fully decommissioned (v1+v2 removed from hf-pdocker-01; voice.hobbs.farm route removed). See memory friday-voice.
• [COMPOSE] FRIDAY Voice stack (all on .150 Docker Desktop, via docker.exe -c desktop-linux):
  • voicecoach-speaches-gpu: :8000 — GPU Whisper large-v3-turbo (STT) + Kokoro-82M af_nova (TTS), CUDA float16. Compose: compose/gpu-machine/voicecoach-speaches/.
  • hermes: nousresearch/hermes-agent:latest gateway — FRIDAY's reasoning agent. Settings transferred from WSL ~/.hermes into named volume hermes-data. Compose: compose/gpu-machine/hermes/.
  • friday-voice: :8320 — push-to-talk console. Calls speaches (host.docker.internal:8000) + hermes (docker exec hermes hermes -Q via wrapper). Public: coach.hobbs.farm → :8320. Compose: compose/gpu-machine/friday-voice/.
• Ollama: :11434 (qwen2.5:14b, Hermes-4.3-36B-GGUF, others)

Port Conflict Notes

Port conflicts already resolved:

• Port 8000: Freed (MCPO moved to 8001, Portainer moved elsewhere)
• Port 3000: AdGuard Home (Gitea moved to 3003)
• Port 3001: Grafana (Semaphore moved to 3004)
• Port 5000: Frigate uses host networking (registry container stopped)
• AI Agent ports re-spaced (2026-03-15): openclaw 2026.3.13 uses gateway+2 and gateway+3 for browser/chrome-relay. Agents spaced 5 apart: margaret=18790, spirit=18795

Critical Configuration Preservation

Encryption Keys & Passwords

• Semaphore: SEMAPHORE_ACCESS_KEY_ENCRYPTION encrypts sensitive data
• Always check for existing encryption keys before generating new ones

Database Version Matching

Match the PostgreSQL/MySQL version to existing data directory format. Check logs for version mismatch errors.

Post-Change Checklist (MANDATORY)

After ANY infrastructure change (deploy, remove, reconfigure, add service, change ports, modify Traefik routes), you MUST complete ALL of these before reporting done:

1. CLAUDE.md — Update service lists, ports, node assignments if changed
2. MEMORY.md — Add/update relevant memory entries
3. Prometheus probes — Add/remove blackbox targets, hot-reload Prometheus
4. Wiki — Launch documentation-keeper agent to update wiki pages (run in background)

Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.

Common Operations

Deploy/Update a Compose Service

1. Edit compose/{node}/{service}/docker-compose.yml
2. SSH to the target node and run:

   cd /mnt/docker-data/compose/{node}/{service}
   docker compose up -d
   

Troubleshoot

# On target node:
docker compose -f /mnt/docker-data/compose/{node}/{service}/docker-compose.yml logs --tail 100
docker ps --format "table {{.Names}}\t{{.Status}}"

Check Cluster Health

# On each node:
docker ps                         # Local containers

Data Management

Storage

All persistent data on NAS at /mnt/docker-data/:

• configs/ - Application configurations
• databases/ - Database volumes
• media/ - Frigate recordings
• traefik/ - SSL certificates

NAS Dependency

Risk: NAS failure = all services fail (single point of failure)

• NFS mount uses soft,timeo=300 for graceful timeout
• Synology should have "Restart automatically after power failure" enabled
• Original data backups remain on local /data/configs/ on hf-pdocker-01

Specialized Agents

vault-credential-manager

Use for: Storing/retrieving credentials from Vaultwarden

documentation-keeper

Use for: Creating/updating documentation in Wiki.js

Known Issues & Workarounds

1. NFS root_squash: Database containers need user: "1024:100" to avoid chown failures on NAS volumes
2. Telegraf Docker Input: Disabled due to Docker socket permission issues across nodes
3. Watchtower API Version: Requires DOCKER_API_VERSION=1.44
4. InfluxDB Connection: Telegraf may show timeout errors if InfluxDB is not running
5. Frigate cameras: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
6. Registry container: Stopped on hf-pdocker-01 to free port 5000 for Frigate
7. systemd-resolved DNSStubListener: Must be disabled on any node running AdGuard Home (port 53 conflict). Fix: /etc/systemd/resolved.conf.d/nostub.conf with DNSStubListener=no, then systemctl restart systemd-resolved. Applied on hf-pdocker-01 and hf-pdocker-02.
8. NFS boot-mount race (hf-pdocker-01): NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s to both fstab entries. If mounts are missing after reboot: sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure.
9. Guacamole schema: App image pinned to guacamole/guacamole:1.6.0 + guacd:1.6.0 with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
10. Gitea image: Pinned to gitea/gitea:1.26.1 with Watchtower disabled (2026-05-20). Watchtower previously pulled broken :latest images causing missing git binary errors.
11. Home Assistant reverse proxy: Config at /mnt/docker-data/configs/homeassistant/configuration.yaml (no hyphen). Must have http: use_x_forwarded_for: true + trusted_proxies: [172.16.0.0/12, 127.0.0.1] or proxied requests return 400.
12. hf-pdocker-01 memory pressure (2026-07-05): Node runs hot (~15/23GB, chronic swap). Mitigations applied: vm.swappiness lowered 60→10 (persisted in /etc/sysctl.d/99-swappiness.conf), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer (~9GB/12GB). Frigate has no semantic-search/face/LPR features enabled.
13. Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05): Both fight over \\.\pipe\docker_engine; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory docker-desktop-unifi-podman-conflict.
14. MySQL InnoDB corruption on NFS: semaphore-db (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via --innodb-force-recovery=1 → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with semaphore user add. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory semaphore-db-nfs-corruption.
15. litellm hangs after DB blip: litellm does NOT self-recover from a transient litellm-db disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual docker restart litellm. Its healthcheck was fixed from curl (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).