hobbs-iac/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository Overview

This repository manages a homelab infrastructure with self-hosted services running across 3 nodes using a hybrid orchestration model: Docker Swarm for core infrastructure services, standalone Docker Compose for isolated services.

Cluster Architecture

Nodes:

• hf-pdocker-01 (192.168.86.192) - Swarm Manager, core infrastructure + Frigate NVR - RESTORED (2026-03-08)
• hf-pdocker-02 (192.168.86.100) - Standalone Compose services (dev tools, utilities)
• bart (192.168.86.167) - Standalone Compose services (media stack, OpenVAS)

Synology NAS (192.168.86.30) - Centralized NFS storage for all service data

Access: Username glyptic, password &Rip8a115 (has sudo access on all nodes)

Service Distribution

hf-pdocker-01 (Manager, 23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
├── [COMPOSE] Frigate NVR (:5000) -- OpenVINO GPU detector, 13 cameras active
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
├── [COMPOSE] clip-service (:5200) -- CLIP embeddings (ViT-L-14, CPU)
├── [COMPOSE] Open WebUI (:3010) -- AI chat UI (Ollama + Anthropic backends)
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] ESPHome (:6052)
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Synapse/Matrix (:8008) + synapse-db (postgres) + wellknown (:8010)
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops], clippy:18800, bob:18810, travis:18815)
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
├── [COMPOSE] AdGuard Home (:3000, :53)
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
├── [COMPOSE] Farm Agent (:8889)
├── [SWARM] Telegraf (global), Node-exporter (global)
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (daily 4am)
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)

bart (Worker, 15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr)
├── [COMPOSE] OpenVAS (vulnerability scanner)
├── [COMPOSE] Onyx (:3005) -- AI document search/RAG (Ollama backend: llava:13b on 192.168.86.196)
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Watchtower
├── [SWARM] Telegraf (global), Node-exporter (global)

hf-pdocker-02 (Worker, 7.6GB RAM, i7-6700T, HDD)
├── [COMPOSE] Unifi + unifi-db (MongoDB)
├── [COMPOSE] Gitea + gitea-db (PostgreSQL)
├── [COMPOSE] Semaphore + semaphore-db (MySQL)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy
├── [COMPOSE] SearXNG (:8888) -- self-hosted meta-search engine (Onyx web search backend)
├── [COMPOSE] Unstructured API (:8000) -- document parsing for Onyx file ingestion
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- Clippy workflow automation
├── [COMPOSE] NetClaw/Aaron (:1179) -- AI network agent
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Watchtower
├── [SWARM] Telegraf (global), Node-exporter (global)

Working with This Repository

Environment Context

You are working in a WSL 2 environment on Windows. Docker is NOT directly available in WSL - all Docker operations must be performed by connecting to the remote nodes via SSH.

Critical: Use the docker-swarm-ops agent (via Task tool) for ALL Docker operations. Do not attempt to run docker commands directly in bash - they will fail.

Storage Architecture

All service data lives on the Synology NAS via NFS, mounted at /mnt/docker-data on all 3 nodes.

/mnt/docker-data/ (NAS: 192.168.86.30:/volume1/Docker)
├── configs/          # Application configs
├── databases/        # Database data directories
├── media/            # Frigate recordings
├── traefik/          # Let's Encrypt certs
└── compose/          # Standalone compose files per node
    ├── hf-pdocker-01/{frigate,frigate-notify,clip-service,matrix,smarthome,guacamole,ai-agents,traefik,adguard}/
    ├── hf-pdocker-02/{unifi,gitea,semaphore,wikijs,vaultwarden,actual-budget,mcpo,trivy,netclaw,watchtower}/
    └── bart/{media,openvas,watchtower}/

NFS mount (all nodes): 192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3 0 0

NFS + Databases: Database containers on NFS require user: "1024:100" to avoid chown failures from NFS root_squash. UID 1024 is the NAS anonymous user.

Repository File Structure

stacks/              # Swarm stack definitions (deployed on hf-pdocker-01)
├── {service-name}/
│   ├── docker-compose.yml
│   └── .env (if needed)

compose/             # Standalone Compose definitions (deployed per-node)
├── hf-pdocker-01/   # Frigate
├── hf-pdocker-02/   # Dev tools & utilities
│   ├── {service}/docker-compose.yml
│   └── {service}/.env
└── bart/            # Media stack & OpenVAS

Deployment Workflow

Swarm services (stacks/):

1. Edit stack file in stacks/{service-name}/docker-compose.yml
2. SCP to hf-pdocker-01 and redeploy: docker stack deploy -c <file> <stack-name>

Standalone Compose services (compose/):

1. Edit compose file in compose/{node}/{service}/docker-compose.yml
2. SCP to NAS path or node, then: cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d

Service Architecture Patterns

Network Patterns:

• Swarm services: Overlay networks for inter-service communication
• Compose services: Bridge networks for service isolation
• Public services (Traefik): Connect to traefik-public overlay + Traefik labels

Database Patterns:

• PostgreSQL: postgres:15-alpine or postgres:16-alpine
• MySQL: mysql:8.0 (Guacamole, Semaphore)
• MongoDB: mongo:7.0 (Unifi)
• Always add com.centurylinklabs.watchtower.enable=false label
• Always add user: "1024:100" for NFS-backed database volumes

Volume Patterns:

• Application data: /mnt/docker-data/configs/{service-name}/
• Database data: /mnt/docker-data/databases/{service-name}-db/
• Traefik certs: /mnt/docker-data/traefik/letsencrypt/

Key Services & Access

Public Services (via Traefik on hf-pdocker-01)

• Matrix: mess.hobbs.farm
• Guacamole: solo.hobbs.farm
• Home Assistant: yoda.hobbs.farm
• Vaultwarden: vault.hobbs.farm
• Gitea: git.hobbs.farm
• Onyx: onyx.hobbs.farm
• Open WebUI: open.hobbs.farm

Internal Services by Node

hf-pdocker-01 (192.168.86.192):

• Frigate: :5000
• Frigate Alerts (v2): :5199 (web UI + REST API)
• clip-service: :5200 (CLIP embeddings, ViT-L-14, CPU)
• Open WebUI: :3010 (AI chat UI, public at open.hobbs.farm)
• Home Assistant: :8123
• ESPHome: :6052
• Mosquitto: :1883
• Matrix/Synapse: :8008
• Well-known (Matrix federation): :8010
• Guacamole: :8199
• AdGuard Home: :3000 (DNS: :53)
• Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
• Grafana: :3001
• Prometheus: :9095
• Alertmanager: :9093
• InfluxDB: :8086
• Blackbox Exporter: :9115
• matrix-alertmanager-receiver: :9094
• AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts], clippy :18800, bob :18810, travis :18815) -- LAN-accessible (bind=lan, 0.0.0.0)
• Farm Manager: :8888 (dashboard + API)
• Farm Agent: :8889

hf-pdocker-02 (192.168.86.100):

• Farm Agent: :8889
• Unifi: :8443
• Gitea: :3003 (web), :2222 (SSH)
• Semaphore: :3004
• Wiki.js: :3002
• Vaultwarden: :26226
• Actual Budget: :5006
• MCPO: :8001
• Trivy: :4954
• SearXNG: :8888 (meta-search, Onyx web search backend)
• Unstructured API: :8000 (document parsing for Onyx file ingestion)
• NetClaw/Aaron: :1179 (internal only)

bart (192.168.86.167):

• Farm Agent: :8889
• Jellyfin: :8096
• Sonarr: :8989
• Radarr: :7878
• SABnzbd: :8082
• NZBHydra2: :5076
• Jellyseerr: :5055
• OpenVAS: :9392
• Onyx: :3005 (AI document search/RAG)

GPU Machine (192.168.86.196, RTX 3090):

• Ollama: :11434 (vision LLM, llava:13b)

Port Conflict Notes

Port conflicts already resolved:

• Port 8000: Now used by Unstructured API on hf-pdocker-02 (MCPO moved to 8001, Portainer moved elsewhere)
• Port 3000: AdGuard Home (Gitea moved to 3003)
• Port 3001: Grafana (Semaphore moved to 3004)
• Port 5000: Frigate uses host networking (registry container stopped)
• AI Agent ports re-spaced (2026-03-15): openclaw 2026.3.13 uses gateway+2 and gateway+3 for browser/chrome-relay. Agents now spaced 5 apart: margaret=18790, spirit=18795, clippy=18800, bob=18810, travis=18815

Critical Configuration Preservation

Encryption Keys & Passwords

• Semaphore: SEMAPHORE_ACCESS_KEY_ENCRYPTION encrypts sensitive data
• Always check for existing encryption keys before generating new ones

Database Version Matching

Match the PostgreSQL/MySQL version to existing data directory format. Check logs for version mismatch errors.

Post-Change Checklist (MANDATORY)

After ANY infrastructure change (deploy, remove, reconfigure, add service, change ports, modify Traefik routes), you MUST complete ALL of these before reporting done:

1. CLAUDE.md — Update service lists, ports, node assignments if changed
2. MEMORY.md — Add/update relevant memory entries
3. Prometheus probes — Add/remove blackbox targets, hot-reload Prometheus
4. Wiki — Launch documentation-keeper agent to update wiki pages (run in background)

Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.

Common Operations

Deploy/Update a Swarm Service

1. Edit stacks/{service}/docker-compose.yml
2. Use docker-swarm-ops agent to SCP and deploy:

   docker stack deploy -c <compose-file> <stack-name>
   

Deploy/Update a Compose Service

1. Edit compose/{node}/{service}/docker-compose.yml
2. Use docker-swarm-ops agent to:

   # On the target node:
   cd /mnt/docker-data/compose/{node}/{service}
   docker compose up -d
   

Troubleshoot

# Swarm services (on hf-pdocker-01):
docker service ls
docker service logs {stack}_{service} --tail 100

# Compose services (on target node):
docker compose -f /mnt/docker-data/compose/{node}/{service}/docker-compose.yml logs --tail 100
docker ps --format "table {{.Names}}\t{{.Status}}"

Check Cluster Health

# On hf-pdocker-01:
docker node ls                    # Swarm node status
docker service ls                 # Swarm services
docker stack ls                   # Deployed stacks

# On each node:
docker ps                         # Local containers (Compose + Swarm)

Data Management

Storage

All persistent data on NAS at /mnt/docker-data/:

• configs/ - Application configurations
• databases/ - Database volumes
• media/ - Frigate recordings
• traefik/ - SSL certificates

NAS Dependency

Risk: NAS failure = all services fail (single point of failure)

• NFS mount uses soft,timeo=300 for graceful timeout
• Synology should have "Restart automatically after power failure" enabled
• Original data backups remain on local /data/configs/ on hf-pdocker-01

Specialized Agents

docker-swarm-ops

Use for: All Docker operations (Swarm + Compose) on any node

vault-credential-manager

Use for: Storing/retrieving credentials from Vaultwarden

documentation-keeper

Use for: Creating/updating documentation in Wiki.js

Known Issues & Workarounds

1. NFS root_squash: Database containers need user: "1024:100" to avoid chown failures on NAS volumes
2. Telegraf Docker Input: Disabled due to Docker socket permission issues across nodes
3. Watchtower API Version: Requires DOCKER_API_VERSION=1.44
4. InfluxDB Connection: Telegraf may show timeout errors if InfluxDB is not running
5. Frigate cameras: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
6. Registry container: Stopped on hf-pdocker-01 to free port 5000 for Frigate
7. systemd-resolved DNSStubListener: Must be disabled on any node running AdGuard Home (port 53 conflict). Fix: /etc/systemd/resolved.conf.d/nostub.conf with DNSStubListener=no, then systemctl restart systemd-resolved. Applied on hf-pdocker-01 and hf-pdocker-02.