jeremy
06e9db70d2
Deploys ghcr.io/lemker/unifi-os-server replacing linuxserver/unifi-network-application. Data dirs moved to /var/lib/unifi-os/ locally (NFS root_squash blocks chown needed by postgres/mongodb/nginx init). Only /persistent remains on NAS for config backups. Access: https://192.168.86.100:11443 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
316 lines
15 KiB
Markdown
316 lines
15 KiB
Markdown
# CLAUDE.md
|
|
|
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
|
|
|
## Repository Overview
|
|
|
|
This repository manages a **homelab infrastructure** with self-hosted services running across 3 nodes using **standalone Docker Compose** on all nodes. (Docker Swarm was fully dismantled on 2026-04-01.)
|
|
|
|
### Cluster Architecture
|
|
|
|
**Nodes:**
|
|
- **hf-pdocker-01** (192.168.86.192) - Core infrastructure + Frigate NVR - **RESTORED (2026-03-08)**
|
|
- **hf-pdocker-02** (192.168.86.100) - Dev tools, utilities
|
|
- **bart** (192.168.86.167) - Media stack, OpenVAS
|
|
|
|
**Synology NAS** (192.168.86.30) - Centralized NFS storage for all service data
|
|
|
|
**Access:** Username `glyptic`, password `&Rip8a115` (has sudo access on all nodes)
|
|
|
|
### Service Distribution
|
|
|
|
```
|
|
hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
|
|
├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
|
|
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
|
|
├── [COMPOSE] clip-service (:5200) -- CLIP embeddings (ViT-L-14, CPU)
|
|
├── [COMPOSE] Home Assistant (:8123)
|
|
├── [COMPOSE] ESPHome (:6052)
|
|
├── [COMPOSE] Mosquitto (:1883, :8883)
|
|
├── [COMPOSE] Synapse/Matrix (:8008) + synapse-db (postgres) + wellknown (:8010)
|
|
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
|
|
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
|
|
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
|
|
├── [COMPOSE] AdGuard Home (:3000, :53)
|
|
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
|
|
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
|
|
├── [COMPOSE] Farm Agent (:8889)
|
|
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
|
|
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.27) 2026-04-04, now Ubuntu
|
|
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
|
|
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
|
|
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (daily 4am)
|
|
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)
|
|
|
|
bart (15GB RAM, i5-4570T, HDD)
|
|
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr)
|
|
├── [COMPOSE] Onyx (:3005) -- redeployed 2026-05-20, all services :latest, Watchtower disabled on all 10
|
|
├── [COMPOSE] OpenVAS (vulnerability scanner)
|
|
├── [COMPOSE] Farm Agent (:8889)
|
|
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
|
|
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
|
|
├── [COMPOSE] Watchtower
|
|
|
|
GPU workstation / Jeremy's PC (192.168.86.27, Ubuntu 24.04 — migrated from Windows 2026-05-28)
|
|
├── RTX 3090 (24GB VRAM), i7-13700KF, 32GB RAM
|
|
├── [COMPOSE] Ollama (:11434) -- 21 local models, v0.24.0
|
|
├── [COMPOSE] Open WebUI (:8080) -- AI chat, points to local Ollama
|
|
├── [REMOVED] vLLM-Hermes -- removed 2026-05-29, VRAM freed for Speaches CUDA
|
|
├── [COMPOSE] Speaches (:8000) -- STT/TTS via faster-whisper + Kokoro, CUDA
|
|
├── [COMPOSE] LiveKit (:7880/:7881) -- WebRTC for VoiceCoach, key: devkey
|
|
├── [COMPOSE] Portainer (:9000, :8002)
|
|
├── [COMPOSE] Prometheus (:9090), DCGM-exporter (:9400), node-exporter (:9100)
|
|
├── [COMPOSE] Watchtower (label-enable mode)
|
|
├── Compose file: /home/glyptic/Docker/compose/docker-compose.yml
|
|
├── vLLM compose: /home/glyptic/Docker/vllm-hermes/docker-compose.yml
|
|
└── Model cache: /home/glyptic/Docker/data/models/
|
|
|
|
hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
|
|
├── [COMPOSE] UniFi OS Server (:11443 HTTPS, :8080 inform, :3478/udp STUN) -- ghcr.io/lemker/unifi-os-server v5.1.15, replaced legacy linuxserver/unifi 2026-05-31. Data local at /var/lib/unifi-os/, /persistent on NAS.
|
|
├── [COMPOSE] Gitea + gitea-db (PostgreSQL)
|
|
├── [COMPOSE] Semaphore + semaphore-db (MySQL)
|
|
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
|
|
├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy
|
|
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
|
|
├── [COMPOSE] Farm Agent (:8889)
|
|
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
|
|
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
|
|
├── [COMPOSE] Watchtower
|
|
|
|
```
|
|
|
|
## Working with This Repository
|
|
|
|
### Environment Context
|
|
|
|
You are working on a **native Ubuntu 24.04 workstation** (previously WSL2 on Windows — migrated 2026-05-28). This machine IS the GPU inference box (RTX 3090, 192.168.86.27). Docker is available locally and runs the inference stack. All homelab node operations (hf-pdocker-01, hf-pdocker-02, bart) still require SSH.
|
|
|
|
**Local Docker on this machine** runs: Ollama, Open WebUI, vLLM-Hermes, Speaches, LiveKit, Portainer, Prometheus, DCGM-exporter.
|
|
|
|
**Critical:** For homelab nodes, use the `docker-swarm-ops` agent via SSH. For local inference services, Docker commands run directly.
|
|
|
|
### Storage Architecture
|
|
|
|
All service data lives on the **Synology NAS** via NFS, mounted at `/mnt/docker-data` on all 3 nodes.
|
|
|
|
```
|
|
/mnt/docker-data/ (NAS: 192.168.86.30:/volume1/Docker)
|
|
├── configs/ # Application configs
|
|
├── databases/ # Database data directories
|
|
├── media/ # Frigate recordings
|
|
├── traefik/ # Let's Encrypt certs
|
|
└── compose/ # Standalone compose files per node
|
|
├── hf-pdocker-01/{frigate,frigate-notify,clip-service,matrix,smarthome,guacamole,ai-agents,traefik,adguard}/
|
|
├── hf-pdocker-02/{unifi,gitea,semaphore,wikijs,vaultwarden,actual-budget,mcpo,trivy,watchtower}/
|
|
└── bart/{media,openvas,watchtower}/
|
|
```
|
|
|
|
**NFS mount (all nodes):** `192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3,nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s 0 0`
|
|
**Note (hf-pdocker-01):** Both NFS mounts hardened with `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` on 2026-05-20 to prevent boot-time race (network not up when NFS tried to mount, cascading ~10 container failures).
|
|
|
|
**NFS + Databases:** Database containers on NFS require `user: "1024:100"` to avoid `chown` failures from NFS root_squash. UID 1024 is the NAS anonymous user.
|
|
|
|
### Repository File Structure
|
|
|
|
```
|
|
stacks/ # LEGACY Swarm stack definitions (Swarm dismantled 2026-04-01, kept for reference)
|
|
|
|
compose/ # Standalone Compose definitions (deployed per-node)
|
|
├── hf-pdocker-01/ # Core infrastructure + Frigate
|
|
├── hf-pdocker-02/ # Dev tools & utilities
|
|
│ ├── {service}/docker-compose.yml
|
|
│ └── {service}/.env
|
|
└── bart/ # Media stack & OpenVAS
|
|
```
|
|
|
|
### Deployment Workflow
|
|
|
|
**All services (compose/):**
|
|
1. Edit compose file in `compose/{node}/{service}/docker-compose.yml`
|
|
2. SCP to NAS path or node, then: `cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d`
|
|
|
|
### Service Architecture Patterns
|
|
|
|
**Network Patterns:**
|
|
- **Compose services**: Bridge networks for service isolation
|
|
- **Public services** (Traefik): Traefik labels + dynamic config files in `/mnt/docker-data/traefik/dynamic/`
|
|
|
|
**Database Patterns:**
|
|
- PostgreSQL: `postgres:15-alpine` or `postgres:16-alpine`
|
|
- MySQL: `mysql:8.0` (Guacamole, Semaphore)
|
|
- MongoDB: `mongo:7.0` (Unifi)
|
|
- **Always** add `com.centurylinklabs.watchtower.enable=false` label
|
|
- **Always** add `user: "1024:100"` for NFS-backed database volumes
|
|
|
|
**Volume Patterns:**
|
|
- Application data: `/mnt/docker-data/configs/{service-name}/`
|
|
- Database data: `/mnt/docker-data/databases/{service-name}-db/`
|
|
- Traefik certs: `/mnt/docker-data/traefik/letsencrypt/`
|
|
|
|
## Key Services & Access
|
|
|
|
### Public Services (via Traefik on hf-pdocker-01)
|
|
|
|
- Matrix: `mess.hobbs.farm`
|
|
- Guacamole: `solo.hobbs.farm`
|
|
- Home Assistant: `yoda.hobbs.farm`
|
|
- Vaultwarden: `vault.hobbs.farm`
|
|
- Gitea: `git.hobbs.farm`
|
|
- VoiceCoach: `coach.hobbs.farm` (LiveKit WebRTC: `voice.hobbs.farm`)
|
|
- Rongorongo: `rongorongo.hobbs.farm`
|
|
|
|
### Internal Services by Node
|
|
|
|
**hf-pdocker-01 (192.168.86.192):**
|
|
- Frigate: :5000
|
|
- Frigate Alerts (v2): :5199 (web UI + REST API)
|
|
- clip-service: :5200 (CLIP embeddings, ViT-L-14, CPU)
|
|
- Home Assistant: :8123
|
|
- ESPHome: :6052
|
|
- Mosquitto: :1883
|
|
- Matrix/Synapse: :8008
|
|
- Well-known (Matrix federation): :8010
|
|
- Guacamole: :8199
|
|
- AdGuard Home: :3000 (DNS: :53)
|
|
- Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
|
|
- Grafana: :3001
|
|
- Prometheus: :9095
|
|
- Alertmanager: :9093
|
|
- InfluxDB: :8086
|
|
- Blackbox Exporter: :9115
|
|
- matrix-alertmanager-receiver: :9094
|
|
- AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0)
|
|
- Farm Manager: :8888 (dashboard + API)
|
|
- Farm Agent: :8889
|
|
- Rongorongo: :8181 (rongorongo-web nginx; reverse-proxies /api/* to rongorongo-api Flask backend on the internal `rongorongo` bridge net). State + images at /mnt/docker-data/configs/rongorongo/data/. Public URL gated by Traefik basicAuth (creds in Vaultwarden).
|
|
- ~~VoiceCoach~~ -- moved to GPU machine (192.168.86.150) 2026-04-04
|
|
|
|
**hf-pdocker-02 (192.168.86.100):**
|
|
- Farm Agent: :8889
|
|
- Unifi: :8443
|
|
- Gitea: :3003 (web), :2222 (SSH)
|
|
- Semaphore: :3004
|
|
- Wiki.js: :3002
|
|
- Vaultwarden: :26226
|
|
- Actual Budget: :5006
|
|
- MCPO: :8001
|
|
- Trivy: :4954
|
|
|
|
**bart (192.168.86.167):**
|
|
- Farm Agent: :8889
|
|
- Jellyfin: :8096
|
|
- Sonarr: :8989
|
|
- Radarr: :7878
|
|
- SABnzbd: :8082
|
|
- NZBHydra2: :5076
|
|
- Jellyseerr: :5055
|
|
- OpenVAS: :9392
|
|
|
|
**GPU Machine (192.168.86.150, RTX 3090, 15GB RAM):**
|
|
- Ollama: :11434 (llama3.1:8b for VoiceCoach LLM, qwen2.5:14b, others)
|
|
- [COMPOSE] VoiceCoach (all services co-located):
|
|
- voicecoach-livekit: :7880/:7881, UDP 50000-50100 (public at voice.hobbs.farm)
|
|
- voicecoach-speaches-gpu: :8000 (GPU Whisper STT + Kokoro TTS, CUDA float16)
|
|
- voicecoach-speechbrain: :8001 (emotion analysis)
|
|
- voicecoach-token-server: :8080 (token + settings API)
|
|
- voicecoach-pipecat: pipeline agent (LLM via local Ollama llama3.1:8b)
|
|
- voicecoach-web: :8085 (public at coach.hobbs.farm)
|
|
- voicecoach-openclaw: :3100 (idle, kept for future Claude API routing)
|
|
- Compose: ~/voicecoach-speaches/docker-compose.yml
|
|
|
|
### Port Conflict Notes
|
|
|
|
Port conflicts already resolved:
|
|
- Port 8000: Freed (MCPO moved to 8001, Portainer moved elsewhere)
|
|
- Port 3000: AdGuard Home (Gitea moved to 3003)
|
|
- Port 3001: Grafana (Semaphore moved to 3004)
|
|
- Port 5000: Frigate uses host networking (registry container stopped)
|
|
- AI Agent ports re-spaced (2026-03-15): openclaw 2026.3.13 uses gateway+2 and gateway+3 for browser/chrome-relay. Agents spaced 5 apart: margaret=18790, spirit=18795
|
|
|
|
## Critical Configuration Preservation
|
|
|
|
### Encryption Keys & Passwords
|
|
|
|
- Semaphore: `SEMAPHORE_ACCESS_KEY_ENCRYPTION` encrypts sensitive data
|
|
- **Always** check for existing encryption keys before generating new ones
|
|
|
|
### Database Version Matching
|
|
|
|
Match the PostgreSQL/MySQL version to existing data directory format. Check logs for version mismatch errors.
|
|
|
|
## Post-Change Checklist (MANDATORY)
|
|
|
|
After ANY infrastructure change (deploy, remove, reconfigure, add service, change ports, modify Traefik routes), you MUST complete ALL of these before reporting done:
|
|
|
|
1. **CLAUDE.md** — Update service lists, ports, node assignments if changed
|
|
2. **MEMORY.md** — Add/update relevant memory entries
|
|
3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus
|
|
4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
|
|
|
|
Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.
|
|
|
|
## Common Operations
|
|
|
|
### Deploy/Update a Compose Service
|
|
|
|
1. Edit `compose/{node}/{service}/docker-compose.yml`
|
|
2. SSH to the target node and run:
|
|
```bash
|
|
cd /mnt/docker-data/compose/{node}/{service}
|
|
docker compose up -d
|
|
```
|
|
|
|
### Troubleshoot
|
|
|
|
```bash
|
|
# On target node:
|
|
docker compose -f /mnt/docker-data/compose/{node}/{service}/docker-compose.yml logs --tail 100
|
|
docker ps --format "table {{.Names}}\t{{.Status}}"
|
|
```
|
|
|
|
### Check Cluster Health
|
|
|
|
```bash
|
|
# On each node:
|
|
docker ps # Local containers
|
|
```
|
|
|
|
## Data Management
|
|
|
|
### Storage
|
|
|
|
All persistent data on NAS at `/mnt/docker-data/`:
|
|
- `configs/` - Application configurations
|
|
- `databases/` - Database volumes
|
|
- `media/` - Frigate recordings
|
|
- `traefik/` - SSL certificates
|
|
|
|
### NAS Dependency
|
|
|
|
**Risk**: NAS failure = all services fail (single point of failure)
|
|
- NFS mount uses `soft,timeo=300` for graceful timeout
|
|
- Synology should have "Restart automatically after power failure" enabled
|
|
- Original data backups remain on local `/data/configs/` on hf-pdocker-01
|
|
|
|
## Specialized Agents
|
|
|
|
### vault-credential-manager
|
|
**Use for:** Storing/retrieving credentials from Vaultwarden
|
|
|
|
### documentation-keeper
|
|
**Use for:** Creating/updating documentation in Wiki.js
|
|
|
|
## Known Issues & Workarounds
|
|
|
|
1. **NFS root_squash**: Database containers need `user: "1024:100"` to avoid chown failures on NAS volumes
|
|
2. **Telegraf Docker Input**: Disabled due to Docker socket permission issues across nodes
|
|
3. **Watchtower API Version**: Requires `DOCKER_API_VERSION=1.44`
|
|
4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running
|
|
5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
|
|
6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
|
|
7. **systemd-resolved DNSStubListener**: Must be disabled on any node running AdGuard Home (port 53 conflict). Fix: `/etc/systemd/resolved.conf.d/nostub.conf` with `DNSStubListener=no`, then `systemctl restart systemd-resolved`. Applied on hf-pdocker-01 and hf-pdocker-02.
|
|
8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`.
|
|
9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
|
|
10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
|
|
11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
|