hobbs-iac/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

This repository manages a **homelab infrastructure** with self-hosted services running across 3 nodes using a **hybrid orchestration** model: Docker Swarm for core infrastructure services, standalone Docker Compose for isolated services.

### Cluster Architecture

**Nodes:**
- **hf-pdocker-01** (192.168.86.192) - Swarm Manager, core infrastructure + Frigate NVR - **RESTORED (2026-03-08)**
- **hf-pdocker-02** (192.168.86.100) - Standalone Compose services (dev tools, utilities)
- **bart** (192.168.86.167) - Standalone Compose services (media stack, OpenVAS)

**Synology NAS** (192.168.86.30) - Centralized NFS storage for all service data

**Access:** Username `glyptic`, password `&Rip8a115` (has sudo access on all nodes)

### Service Distribution

```
hf-pdocker-01 (Manager, 23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
├── [COMPOSE] Frigate NVR (:5000) -- OpenVINO GPU detector, 13 cameras active
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
├── [COMPOSE] clip-service (:5200) -- CLIP embeddings (ViT-L-14, CPU)
├── [COMPOSE] Open WebUI (:3010) -- AI chat UI (Ollama + Anthropic backends)
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] ESPHome (:6052)
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Synapse/Matrix (:8008) + synapse-db (postgres) + wellknown (:8010)
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops], clippy:18800, bob:18810, travis:18815)
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
├── [COMPOSE] AdGuard Home (:3000, :53)
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
├── [COMPOSE] Farm Agent (:8889)
├── [SWARM] Telegraf (global), Node-exporter (global)
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (daily 4am)
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)

bart (Worker, 15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr)
├── [COMPOSE] OpenVAS (vulnerability scanner)
├── [COMPOSE] Onyx (:3005) -- AI document search/RAG (Ollama backend: llava:13b on 192.168.86.196)
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Watchtower
├── [SWARM] Telegraf (global), Node-exporter (global)

hf-pdocker-02 (Worker, 7.6GB RAM, i7-6700T, HDD)
├── [COMPOSE] Unifi + unifi-db (MongoDB)
├── [COMPOSE] Gitea + gitea-db (PostgreSQL)
├── [COMPOSE] Semaphore + semaphore-db (MySQL)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy
├── [COMPOSE] SearXNG (:8888) -- self-hosted meta-search engine (Onyx web search backend)
├── [COMPOSE] Unstructured API (:8000) -- document parsing for Onyx file ingestion
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- Clippy workflow automation
├── [COMPOSE] NetClaw/Aaron (:1179) -- AI network agent
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Watchtower
├── [SWARM] Telegraf (global), Node-exporter (global)

```

## Working with This Repository

### Environment Context

You are working in a **WSL 2 environment** on Windows. Docker is NOT directly available in WSL - all Docker operations must be performed by connecting to the remote nodes via SSH.

**Critical:** Use the `docker-swarm-ops` agent (via Task tool) for ALL Docker operations. Do not attempt to run docker commands directly in bash - they will fail.

### Storage Architecture

All service data lives on the **Synology NAS** via NFS, mounted at `/mnt/docker-data` on all 3 nodes.

```
/mnt/docker-data/ (NAS: 192.168.86.30:/volume1/Docker)
├── configs/          # Application configs
├── databases/        # Database data directories
├── media/            # Frigate recordings
├── traefik/          # Let's Encrypt certs
└── compose/          # Standalone compose files per node
    ├── hf-pdocker-01/{frigate,frigate-notify,clip-service,matrix,smarthome,guacamole,ai-agents,traefik,adguard}/
    ├── hf-pdocker-02/{unifi,gitea,semaphore,wikijs,vaultwarden,actual-budget,mcpo,trivy,netclaw,watchtower}/
    └── bart/{media,openvas,watchtower}/
```

**NFS mount (all nodes):** `192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3 0 0`

**NFS + Databases:** Database containers on NFS require `user: "1024:100"` to avoid `chown` failures from NFS root_squash. UID 1024 is the NAS anonymous user.

### Repository File Structure

```
stacks/              # Swarm stack definitions (deployed on hf-pdocker-01)
├── {service-name}/
│   ├── docker-compose.yml
│   └── .env (if needed)

compose/             # Standalone Compose definitions (deployed per-node)
├── hf-pdocker-01/   # Frigate
├── hf-pdocker-02/   # Dev tools & utilities
│   ├── {service}/docker-compose.yml
│   └── {service}/.env
└── bart/            # Media stack & OpenVAS
```

### Deployment Workflow

**Swarm services (stacks/):**
1. Edit stack file in `stacks/{service-name}/docker-compose.yml`
2. SCP to hf-pdocker-01 and redeploy: `docker stack deploy -c <file> <stack-name>`

**Standalone Compose services (compose/):**
1. Edit compose file in `compose/{node}/{service}/docker-compose.yml`
2. SCP to NAS path or node, then: `cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d`

### Service Architecture Patterns

**Network Patterns:**
- **Swarm services**: Overlay networks for inter-service communication
- **Compose services**: Bridge networks for service isolation
- **Public services** (Traefik): Connect to `traefik-public` overlay + Traefik labels

**Database Patterns:**
- PostgreSQL: `postgres:15-alpine` or `postgres:16-alpine`
- MySQL: `mysql:8.0` (Guacamole, Semaphore)
- MongoDB: `mongo:7.0` (Unifi)
- **Always** add `com.centurylinklabs.watchtower.enable=false` label
- **Always** add `user: "1024:100"` for NFS-backed database volumes

**Volume Patterns:**
- Application data: `/mnt/docker-data/configs/{service-name}/`
- Database data: `/mnt/docker-data/databases/{service-name}-db/`
- Traefik certs: `/mnt/docker-data/traefik/letsencrypt/`

## Key Services & Access

### Public Services (via Traefik on hf-pdocker-01)

- Matrix: `mess.hobbs.farm`
- Guacamole: `solo.hobbs.farm`
- Home Assistant: `yoda.hobbs.farm`
- Vaultwarden: `vault.hobbs.farm`
- Gitea: `git.hobbs.farm`
- Onyx: `onyx.hobbs.farm`
- Open WebUI: `open.hobbs.farm`

### Internal Services by Node

**hf-pdocker-01 (192.168.86.192):**
- Frigate: :5000
- Frigate Alerts (v2): :5199 (web UI + REST API)
- clip-service: :5200 (CLIP embeddings, ViT-L-14, CPU)
- Open WebUI: :3010 (AI chat UI, public at open.hobbs.farm)
- Home Assistant: :8123
- ESPHome: :6052
- Mosquitto: :1883
- Matrix/Synapse: :8008
- Well-known (Matrix federation): :8010
- Guacamole: :8199
- AdGuard Home: :3000 (DNS: :53)
- Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
- Grafana: :3001
- Prometheus: :9095
- Alertmanager: :9093
- InfluxDB: :8086
- Blackbox Exporter: :9115
- matrix-alertmanager-receiver: :9094
- AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts], clippy :18800, bob :18810, travis :18815) -- LAN-accessible (bind=lan, 0.0.0.0)
- Farm Manager: :8888 (dashboard + API)
- Farm Agent: :8889

**hf-pdocker-02 (192.168.86.100):**
- Farm Agent: :8889
- Unifi: :8443
- Gitea: :3003 (web), :2222 (SSH)
- Semaphore: :3004
- Wiki.js: :3002
- Vaultwarden: :26226
- Actual Budget: :5006
- MCPO: :8001
- Trivy: :4954
- SearXNG: :8888 (meta-search, Onyx web search backend)
- Unstructured API: :8000 (document parsing for Onyx file ingestion)
- NetClaw/Aaron: :1179 (internal only)

**bart (192.168.86.167):**
- Farm Agent: :8889
- Jellyfin: :8096
- Sonarr: :8989
- Radarr: :7878
- SABnzbd: :8082
- NZBHydra2: :5076
- Jellyseerr: :5055
- OpenVAS: :9392
- Onyx: :3005 (AI document search/RAG)

**GPU Machine (192.168.86.196, RTX 3090):**
- Ollama: :11434 (vision LLM, llava:13b)

### Port Conflict Notes

Port conflicts already resolved:
- Port 8000: Now used by Unstructured API on hf-pdocker-02 (MCPO moved to 8001, Portainer moved elsewhere)
- Port 3000: AdGuard Home (Gitea moved to 3003)
- Port 3001: Grafana (Semaphore moved to 3004)
- Port 5000: Frigate uses host networking (registry container stopped)
- AI Agent ports re-spaced (2026-03-15): openclaw 2026.3.13 uses gateway+2 and gateway+3 for browser/chrome-relay. Agents now spaced 5 apart: margaret=18790, spirit=18795, clippy=18800, bob=18810, travis=18815

## Critical Configuration Preservation

### Encryption Keys & Passwords

- Semaphore: `SEMAPHORE_ACCESS_KEY_ENCRYPTION` encrypts sensitive data
- **Always** check for existing encryption keys before generating new ones

### Database Version Matching

Match the PostgreSQL/MySQL version to existing data directory format. Check logs for version mismatch errors.

## Post-Change Checklist (MANDATORY)

After ANY infrastructure change (deploy, remove, reconfigure, add service, change ports, modify Traefik routes), you MUST complete ALL of these before reporting done:

1. **CLAUDE.md** — Update service lists, ports, node assignments if changed
2. **MEMORY.md** — Add/update relevant memory entries
3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus
4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)

Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.

## Common Operations

### Deploy/Update a Swarm Service

1. Edit `stacks/{service}/docker-compose.yml`
2. Use docker-swarm-ops agent to SCP and deploy:
   ```bash
   docker stack deploy -c <compose-file> <stack-name>
   ```

### Deploy/Update a Compose Service

1. Edit `compose/{node}/{service}/docker-compose.yml`
2. Use docker-swarm-ops agent to:
   ```bash
   # On the target node:
   cd /mnt/docker-data/compose/{node}/{service}
   docker compose up -d
   ```

### Troubleshoot

```bash
# Swarm services (on hf-pdocker-01):
docker service ls
docker service logs {stack}_{service} --tail 100

# Compose services (on target node):
docker compose -f /mnt/docker-data/compose/{node}/{service}/docker-compose.yml logs --tail 100
docker ps --format "table {{.Names}}\t{{.Status}}"
```

### Check Cluster Health

```bash
# On hf-pdocker-01:
docker node ls                    # Swarm node status
docker service ls                 # Swarm services
docker stack ls                   # Deployed stacks

# On each node:
docker ps                         # Local containers (Compose + Swarm)
```

## Data Management

### Storage

All persistent data on NAS at `/mnt/docker-data/`:
- `configs/` - Application configurations
- `databases/` - Database volumes
- `media/` - Frigate recordings
- `traefik/` - SSL certificates

### NAS Dependency

**Risk**: NAS failure = all services fail (single point of failure)
- NFS mount uses `soft,timeo=300` for graceful timeout
- Synology should have "Restart automatically after power failure" enabled
- Original data backups remain on local `/data/configs/` on hf-pdocker-01

## Specialized Agents

### docker-swarm-ops
**Use for:** All Docker operations (Swarm + Compose) on any node

### vault-credential-manager
**Use for:** Storing/retrieving credentials from Vaultwarden

### documentation-keeper
**Use for:** Creating/updating documentation in Wiki.js

## Known Issues & Workarounds

1. **NFS root_squash**: Database containers need `user: "1024:100"` to avoid chown failures on NAS volumes
2. **Telegraf Docker Input**: Disabled due to Docker socket permission issues across nodes
3. **Watchtower API Version**: Requires `DOCKER_API_VERSION=1.44`
4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running
5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
7. **systemd-resolved DNSStubListener**: Must be disabled on any node running AdGuard Home (port 53 conflict). Fix: `/etc/systemd/resolved.conf.d/nostub.conf` with `DNSStubListener=no`, then `systemctl restart systemd-resolved`. Applied on hf-pdocker-01 and hf-pdocker-02.