jeremy/hobbs-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f4f64eb5d45c1188459970b732352205973f97a5
hobbs-iac/CLAUDE.md
T

62 KiB
Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repository Overview

This repository manages a homelab infrastructure with self-hosted services running across 3 nodes using standalone Docker Compose on all nodes. (Docker Swarm was fully dismantled on 2026-04-01.)

Cluster Architecture

Nodes:

  • hf-pdocker-01 (192.168.86.192) - Core infrastructure + Frigate NVR - RESTORED (2026-03-08)
  • hf-pdocker-02 (192.168.86.100) - Dev tools, utilities
  • bart (192.168.86.167) - Media stack, OpenVAS

Synology NAS (192.168.86.30) - Centralized NFS storage for all service data

Access: Username glyptic, password &Rip8a115 (has sudo access on all nodes)

Service Distribution

hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
├── [COMPOSE] clip-service (:5200) -- **RUNNING, and actively used.** CORRECTED 2026-09-21: the 2026-07-13 decommission note below was wrong for the current deployment — it was redeployed and the *live* `frigate-notify/docker-compose.yml` sets `CLIP_SERVICE_URL=http://localhost:5200`, with live `app/recognition.py` calling `/embed`. ~1.2GB RSS. **Do not remove without first rewiring frigate-notify's recognition pipeline.**
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] ESPHome (:6052) -- manages **Pool Value Controls** (ESP8266 at static **192.168.86.6**, drives pool pumps/valves; only ESPHome device registered in HA). Also polelightpond (.14) and living_room_lights (.153) configs, NOT in HA. See Known Issue #39.
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Synapse/Matrix (:8008) + wellknown (:8010) + synapse-db (**postgres:16-alpine, LOCAL /var/lib/synapse-db, C collation**) -- Synapse data on LOCAL /var/lib/synapse. Migrated SQLite->PostgreSQL 2026-08-20 (required by syn2mas). **Auth is delegated to MAS** -- Synapse no longer serves /login.
├── [COMPOSE] **MAS / Matrix Authentication Service (:8020 web, :8021 health) + mas-db (postgres:16-alpine, LOCAL /var/lib/mas-db)** -- OAuth2/OIDC provider for Matrix. Public: auth.hobbs.farm. Deployed 2026-08-20 to enable QR sign-in (MSC4108). QR sign-in **verified working end-to-end 2026-08-20**. **Every Matrix login now depends on this service.**
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops]) -- **OpenClaw 2026.7.1-2, images PINNED via Dockerfile ARG** (`compose/hf-pdocker-01/ai-agents/{margaret,spirit}/Dockerfile`). Both primary `anthropic/claude-sonnet-4-6`, fallback `google/gemini-2.5-flash`.
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090) -- `traefik:v3.7` (3.7.16, bumped from v3.6 2026-10-26; 3.7 migration notes are k8s-only for us)
├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192.
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
├── [COMPOSE] **Onyx (RAG/search) -- 10 containers, `/mnt/docker-data/compose/hf-pdocker-01/onyx`.** NOT decommissioned (the bart entry below refers to an older, separate deployment). Public: **onyx.hobbs.farm** -> nginx :3005. Custom build: `gitea_patch.py` + `unstructured_patch.py` index Gitea (`git.hobbs.farm/api/v1`) and call unstructured-api on .100:8000. LLM -> `192.168.86.150:18020/v1` (`qwen3.8-27b`) — **dead while .150 is down.** `onyx-opensearch-1` **stopped 2026-09-21** (`ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false`, held no Onyx data, nothing `depends_on` it; freed ~1.1GB). **2026-10-10: it came back on its own after .192's crash/reboot (1.5GB) — now stopped, `restart=no`, and gated behind compose `profiles: ["opensearch"]` so neither reboots nor `compose up` revive it.** Restart with `docker compose --profile opensearch up -d opensearch` if the flag is ever flipped.
├── [COMPOSE] hermes -- **moved here from .150** (host network, `/mnt/docker-data/compose/hf-pdocker-01/hermes`, pinned `v2026.8.18`). Points at `http://192.168.86.150:18020/v1` model `qwen3.8-27b`; that endpoint is **dead**, so every call burns 200s x3 retries and fails. Note this is a different host:port than the `.150:8081` llamacpp setup documented in the GPU Machine section.
├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
├── [COMPOSE] synapse-admin -- Synapse admin UI
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (4x daily 4am/10am/4pm/10pm, log ~/frigate-restart.log)
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)

bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, **Seerr** -- replaced Jellyseerr 2026-10-17, `seerr/seerr:v3.5.1`, config LOCAL /var/lib/seerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin on official STABLE 12.x, pinned to the dated tag `jellyfin/jellyfin:12.3.20261023-013714` (12.3.0, 2026-10-26).** History: ran the 2026060106 unstable build from 2026-07-14 because the DB had been migrated to the v12 schema (10.11.x cannot read it); stable 12.0 shipped 2026-09-08, moved to 12.2 on 2026-10-17, 12.3 on 2026-10-26 (DB backup `/var/lib/jellyfin-backup-20261026-pre123`). **Never go below 12.**
├── [REMOVED] Onyx -- this *bart* deployment was decommissioned 2026-07-13. **A different, newer Onyx now runs on hf-pdocker-01 (see above) — 'Onyx is gone' is false.**
├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15.
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Watchtower -- `nickfedor/watchtower` (maintained fork; containrrr is archived), runs **twice daily 04:00 + 16:00** (`WATCHTOWER_SCHEDULE=0 0 4,16 * * *`)

hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
├── [REMOVED] UniFi OS Server -- the `unifi-os-server` container (ghcr.io/lemker/unifi-os-server, v5.1.15) was decommissioned 2026-10-05. It was a **duplicate/idle** controller -- the real UniFi controller is the **Podman UniFi OS Server on the Windows box (.150)**, which manages all UniFi devices (e.g. `downstairsswitch` US-24-250W at 192.168.86.159, inform `http://192.168.86.150:8080/inform`). Container + image + network removed; ports 11443/8080/3478/etc freed on .100. **Data preserved** at `/var/lib/unifi-os/*` (971M local) + `/mnt/docker-data/configs/unifi-os/persistent` (NAS); compose dir kept for reference. **Only .150 manages UniFi, by design.** **2026-10-26: it was found RUNNING again** (the 10-05 removal never stuck -- journal shows it up again later that same day, then recreated from its compose dir 2026-10-17 21:06Z). Removed again (container, `unifi_default` network, `ghcr.io/lemker/unifi-os-server` + stale `linuxserver/unifi-network-application` images) and the compose file renamed to `docker-compose.yml.decommissioned-20261026` + `DECOMMISSIONED.txt` so a stray `compose up` cannot revive it. Data still preserved.
├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
├── [COMPOSE] Vaultwarden, MCPO, Trivy
├── [COMPOSE] SearXNG (`searxng/searxng:latest`) -- found running 2026-10-26, previously undocumented here
├── [COMPOSE] unstructured-api (:8000, `downloads.unstructured.io/unstructured-io/unstructured-api:latest`) -- used by Onyx's `unstructured_patch.py` on .192
├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget)
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
├── [COMPOSE] CouchDB (:5984) -- Obsidian LiveSync backend for the Second Brain vault. LAN-only. LOCAL disk (NOT NFS).
├── [COMPOSE] **Hobbs Family Organizer (:8340)** -- Shield TV family dashboard (calendar/weather/headlines/markets), deployed 2026-11-04, replaces the ChatGPT-Sites-hosted `dayline-tv` page. Next.js 16 / Node 22, image **built locally** (`hobbs-family-organizer:1.2.0` since 2026-11-04 evening; `1.0.0`/`1.1.0` kept for rollback; Watchtower disabled). **v1.2.0 adds the photo frame `/photo`** (slideshow of Google Photos shared albums, scraped from the share page because Google's Library API lost album access 2025-03-31, + admin uploads; synced every 6h into LOCAL `data/photos/`, ~0.5MB/photo; current album "TV screensaver", 46 photos). Photo health `GET /api/health/photos` = 503 when an album has not synced for 48h. **Shield photo app `com.dayline.photos` v1.0** (2026-11-04, separate APK: launcher tile + screensaver/DreamService, both load `/photo`; default URL baked in, override via `files/base_url` with `run-as`). Built on .100 by `compose/hf-pdocker-02/hobbs-organizer/build-photos-apk.sh` (JDK17 container, SDK cached in volume `hobbs-android-sdk`). **Signing key** `/var/lib/hobbs-organizer/apk-signing/` (root 600) + NAS copy `/mnt/docker-data/configs/hobbs-organizer/apk-signing/` -- losing it means uninstall/reinstall for any update. Installed on .186 only (canary), set as its screensaver (previous: `com.google.android.backdrop/.Backdrop`, saved in `~/hobbs-shield/screensaver-before-186.txt` on .100). **v1.1.0 adds `/admin`** (password, LAN-only, manages calendar links with masked display/test/history-revert); settings + scrypt password hash live in LOCAL `/var/lib/hobbs-organizer/data` (uid 10001, 700) — calendars now come from `data/settings.json`, the secrets file is only a first-start seed. Inside the container a small front proxy on :3000 stamps the real client IP (Next.js runs on 127.0.0.1:3001) so admin can refuse non-LAN and proxied requests. Password reset: `docker exec -u 10001 hobbs-family-organizer-organizer-1 node container/reset-admin.mjs`. Source = `tv app/Hobbs-Family-Organizer-Self-Hosted-v1.0.0/`; deploy dir `/mnt/docker-data/compose/hf-pdocker-02/hobbs-organizer` (bundle + `compose.override.yaml` + `.env`, repo copy `compose/hf-pdocker-02/hobbs-organizer/`). **Trusted-LAN mode** (empty token, per user). Secrets on LOCAL `/var/lib/hobbs-organizer/secrets/` (root 600 -- NFS root_squash would block start.mjs). Both Shields (.179, .186) run APK v1.4 pointed at it. See memory `hobbs-family-organizer`.
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Watchtower -- `nickfedor/watchtower` (maintained fork; containrrr is archived), runs **twice daily 04:00 + 16:00** (`WATCHTOWER_SCHEDULE=0 0 4,16 * * *`)

Working with This Repository

Environment Context

You are working in a WSL 2 environment on Windows. Docker is NOT directly available in WSL - all Docker operations must be performed by connecting to the remote nodes via SSH.

Critical: Docker is not available locally in WSL. All Docker operations must be performed by connecting to the remote nodes via SSH.

Storage Architecture

All service data lives on the Synology NAS via NFS, mounted at /mnt/docker-data on all 3 nodes.

/mnt/docker-data/ (NAS: 192.168.86.30:/volume1/Docker)
├── configs/          # Application configs
├── databases/        # Database data directories
├── media/            # Frigate recordings
├── traefik/          # Let's Encrypt certs
└── compose/          # Standalone compose files per node
    ├── hf-pdocker-01/{frigate,frigate-notify,clip-service,matrix,smarthome,guacamole,ai-agents,traefik,adguard}/
    ├── hf-pdocker-02/{unifi,gitea,semaphore,wikijs,vaultwarden,actual-budget,mcpo,trivy,watchtower}/
    └── bart/{media,openvas,watchtower}/

hf-pdocker-02 fixed 2026-11-04 after the 11-03 power event (dockerd was dead 27h: stale Swarm membership made it fatal-exit when .192:2377 was unreachable, and /mnt/docker-data had not mounted): docker swarm leave --force done (side effect: it wiped Docker's embedded-DNS container-name records, so wikijs-db/gitea-db/… stopped resolving even for new containers -- external DNS was fine, existing DB pools kept working, new connections failed. Fixed by restarting all 18 containers DBs-first. When doing bart: restart its containers or dockerd right after swarm leave), and the /mnt/docker-data fstab line now has nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s (backup /etc/fstab.bak-20261104-nfs-nofail). Its other NAS lines (/mnt/infrastructure, /mnt/NAS cifs, /data/media, /mnt/nas-media) still lack it. bart and .192 still report Swarm=active -- .192 is the old manager so it boots fine; bart would fail the same way as .100 did if .192 is down at boot (not yet fixed, user approved only .100).

NFS mount (all nodes): 192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3,nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s 0 0 Note (hf-pdocker-01): Both NFS mounts hardened with nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s on 2026-05-20 to prevent boot-time race (network not up when NFS tried to mount, cascading ~10 container failures).

NFS + Databases: Database containers on NFS require user: "1024:100" to avoid chown failures from NFS root_squash. UID 1024 is the NAS anonymous user.

Repository File Structure

stacks/              # LEGACY Swarm stack definitions (Swarm dismantled 2026-04-01, kept for reference)

compose/             # Standalone Compose definitions (deployed per-node)
├── hf-pdocker-01/   # Core infrastructure + Frigate
├── hf-pdocker-02/   # Dev tools & utilities
│   ├── {service}/docker-compose.yml
│   └── {service}/.env
└── bart/            # Media stack & OpenVAS

Deployment Workflow

All services (compose/):

  1. Edit compose file in compose/{node}/{service}/docker-compose.yml
  2. SCP to NAS path or node, then: cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d

Service Architecture Patterns

Network Patterns:

  • Compose services: Bridge networks for service isolation
  • Public services (Traefik): Traefik labels + dynamic config files in /mnt/docker-data/traefik/dynamic/

Database Patterns:

  • PostgreSQL: postgres:15-alpine or postgres:16-alpine
  • MySQL: mysql:8.0 (Guacamole, Semaphore)
  • MongoDB: mongo:7.0 (Unifi)
  • Always add com.centurylinklabs.watchtower.enable=false label
  • Always add user: "1024:100" for NFS-backed database volumes

Volume Patterns:

  • Application data: /mnt/docker-data/configs/{service-name}/
  • Database data: /mnt/docker-data/databases/{service-name}-db/
  • Traefik certs: /mnt/docker-data/traefik/letsencrypt/

Key Services & Access

Public Services (via Traefik on hf-pdocker-01)

  • Matrix: mess.hobbs.farm
  • Guacamole: solo.hobbs.farm
  • Home Assistant: yoda.hobbs.farm
  • Vaultwarden: vault.hobbs.farm
  • Gitea: git.hobbs.farm
  • FRIDAY Voice: coach.hobbs.farm — DECOMMISSIONED 2026-08-26; coach.hobbs.farm returns 404, which is EXPECTED, not an outage. CORRECTED 2026-09-21: the "UP as of 2026-08-20" text previously here was overtaken by events. The live Prometheus config records the voicecoach teardown (2026-08-24) and the FRIDAY Voice + speaches teardown (2026-08-26) — routing and monitoring sides done, with "container/volume teardown on .150 still pending manual action next Windows boot". The 6-container voicecoach project is indeed gone from .192 (verified 2026-09-21). Its .150 images/volumes may still exist. See memory friday-voice.
  • MAS (Matrix auth): auth.hobbs.farm -- OAuth2/OIDC issuer. If this is down, nobody can log into Matrix (existing tokens keep working).
  • Onyx (RAG/search): onyx.hobbs.farm -- self-registration is LAN-only (Traefik onyx-register-lockdown router). See Known Issue #34.
  • Rongorongo: rongorongo.hobbs.farm
  • LinkedStorm: linkedstorm.hobbs.farm

Internal Services by Node

hf-pdocker-01 (192.168.86.192):

  • Frigate: :5000
  • Frigate Alerts (v2): :5199 (web UI + REST API)
  • Home Assistant: :8123
  • ESPHome: :6052
  • Mosquitto: :1883
  • Matrix/Synapse: :8008
  • MAS: :8020 (web/oauth/compat), :8021 (health only -- /health is NOT on :8020)
  • Well-known (Matrix federation): :8010
  • Guacamole: :8199
  • Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
  • Grafana: :3001
  • Prometheus: :9095
  • Alertmanager: :9093
  • InfluxDB: :8086
  • Blackbox Exporter: :9115
  • matrix-alertmanager-receiver: :9094
  • AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0). OpenClaw 2026.7.1-2 (upgraded 2026-08-20 from 2026.5.27). Matrix: @margaret:hobbs.farm / @spirit:hobbs.farm, token auth against http://192.168.86.192:8008, encryption off, requireMention: false. Rollback images kept as margaret-openclaw:rollback-2026.5.27 / spirit-ops:rollback-2026.5.27.
  • Onyx: :3005 (onyx-nginx). Public onyx.hobbs.farm. onyx-opensearch-1 deliberately stopped 2026-09-21.
  • hermes: no published port (host network) -- see service tree; its LLM endpoint is currently dead.
  • Farm Manager: :8888 (dashboard + API)
  • Farm Agent: :8889
  • Rongorongo: :8181 (rongorongo-web nginx; reverse-proxies /api/* to rongorongo-api Flask backend on the internal rongorongo bridge net). State + images at /mnt/docker-data/configs/rongorongo/data/. Public URL gated by Traefik basicAuth (creds in Vaultwarden).
  • LinkedStorm: :8300 (pipeline/VAD+STT+speaker-ID), :8310 (API/dashboard), cron (no port). Public: linkedstorm.hobbs.farm -> :8310. Source: /opt/linkedstorm (rsync from WSL). Data: /mnt/docker-data/configs/linkedstorm/. STT via Speaches GPU machine :8000, LLM via Ollama :11434 (qwen2.5:14b).
  • VoiceCoach -- replaced by FRIDAY Voice on .150 (see GPU Machine section + memory friday-voice). CORRECTION 2026-08-20: the teardown claimed below never actually happened. All 6 containers of compose project voicecoach (voicecoach-web :8085, -pipecat, -speaches, -speechbrain, -openclaw :3100, -livekit :7880-7881 + 50000-50100/udp) are running on .192, up since ~2026-08-06. Earlier text said both projects were docker compose down'd on 2026-08-05; that is false for voicecoach (voicecoach-v2 is genuinely gone). ~1GB RAM. voicecoach-openclaw was upgraded to OpenClaw 2026.7.1-2 on 2026-08-20 via docker compose up -d --no-deps voicecoach-openclaw (single service only — does not resurrect anything, the rest were already up). Its model is ollama-local/qwen2.5:14b at the dead 192.168.86.27:11434 with no fallbacks, so it cannot serve a turn. Decide: fix its model chain or actually tear the stack down.

hf-pdocker-02 (192.168.86.100):

  • Farm Agent: :8889
  • UniFi OS Server: :11443 -- REMOVED 2026-10-05 (duplicate controller; see service tree). The UniFi controller lives on .150 (Podman), UI at https://192.168.86.150:11443, inform :8080.
  • Gitea: :3003 (web), :2222 (SSH)
  • Semaphore: :3004
  • Wiki.js: :3002
  • Vaultwarden: :26226
  • MCPO: :8001
  • Trivy: :4954
  • CouchDB: :5984 -- Obsidian LiveSync backend. DB secondbrain; admin creds in Vaultwarden ("CouchDB - Obsidian LiveSync"). LAN-only (no Traefik route). Data on LOCAL disk /var/lib/couchdb (NFS breaks CouchDB file locking). local.ini must be mounted READ-WRITE -- with :ro CouchDB exits 1 and logs nothing. Auth required on everything except /_up (left open for the blackbox probe). Vault repo: git.hobbs.farm/jeremy/second-brain (private); vault lives at C:\Users\Jeremy\second-brain on the Windows box.
  • Hobbs Family Organizer: :8340 -- Shield TV dashboard, GET /api/health (JSON, calendarConfigurationValid, adminConfigured). Dashboard open on LAN (trusted-LAN mode); /photo = photo frame; /api/health/photos (503 = album sync stale >48h, probe this); /admin is password-protected and LAN-only (direct 192.168.86.0/24 only -- anything arriving via a proxy gets 404, so do NOT add a Traefik route expecting admin to work through it). No Traefik route.

bart (192.168.86.167):

  • Farm Agent: :8889
  • Jellyfin: :8096
  • Sonarr: :8989
  • Radarr: :7878
  • SABnzbd: :8082
  • NZBHydra2: :5076
  • Jellyseerr: :5055

GPU Machine (192.168.86.150, RTX 3090):

  • NOTE (2026-07-05): .150 is actually the user's Windows workstation (runs the WSL/Claude Code session), not a headless Linux server. It runs Docker Desktop (WSL2 backend, RTX 3090 GPU passthrough working) alongside UniFi OS Server (Podman) — this is the ONE real UniFi controller (the duplicate container on .100 was removed 2026-10-05). UI https://192.168.86.150:11443, inform http://192.168.86.150:8080. Manages all UniFi devices incl. downstairsswitch (US-24-250W, 192.168.86.159; SSH admin/node-password). Docker Desktop panics if UniFi/Podman claims \\.\pipe\docker_engine first — start Docker Desktop before UniFi OS Server. See memory docker-desktop-unifi-podman-conflict.
  • FRIDAY Voice replaced VoiceCoach (2026-07-05). VoiceCoach fully decommissioned (v1+v2 removed from hf-pdocker-01; voice.hobbs.farm route removed). See memory friday-voice.
  • [COMPOSE] FRIDAY Voice stack (all on .150 Docker Desktop, via docker.exe -c desktop-linux):
    • voicecoach-speaches-gpu: :8000 — GPU Whisper large-v3-turbo (STT) + Kokoro-82M af_nova (TTS), CUDA float16. Compose: compose/gpu-machine/voicecoach-speaches/. REMOVED 2026-08-21 (see the speaches note at the end of this section) — rebuild when voice I/O is wanted again.
    • hermes: PINNED nousresearch/hermes-agent:v2026.8.18 (upgraded 2026-08-20 from v2026.7.1 / v0.18.0 → v0.20.4). FRIDAY's reasoning agent. Settings in named volume hermes-data. Compose: compose/gpu-machine/hermes/. Was on :latest with Watchtower disabled, so nothing ever re-pulled it — it sat 7 weeks / 9 releases behind. Note :latest on this repo is the same digest as main (rolling dev build), so pin a vYYYY.M.D tag. Rollback image nousresearch/hermes-agent:rollback-2026.7.1; volume backup in volume hermes-data-backup-20260820 (hermes-data-20260820.tgz, 29MB). Matrix pairing: hermes denies unknown senders by default — approve with docker exec hermes hermes pairing approve matrix <CODE> (@jeremy:hobbs.farm approved 2026-08-20; state persists in the volume). RESOLVED 2026-08-21 by moving off ChatGPT entirely. Its Codex refresh token had expired (Could not validate your refresh token) and, with no API key set for any provider, that was the only auth path — both Matrix and /api/ask failed. Verified NOT an upgrade regression (the rollback image failed identically on the same volume). Rather than re-auth interactively, hermes was pointed at the local llamacpp endpoint: model.provider=llamacpp, model.default=qwen3.8-64k, model.base_url=http://192.168.86.150:8081/v1. No tokens, no TTY, no subscription. Config backups: /opt/data/config.yaml.bak-20260820-ollama and .bak-20260821-llamacpp. hermes model is interactive-only — use hermes config set for non-interactive changes. hermes also hard-refuses any model advertising <64K context ("Choose a model with at least 64K context"), which is why the served model is a 64k build, not 32k.
    • llamacpp: :8081 — the inference backend for hermes as of 2026-08-21. ghcr.io/ggml-org/llama.cpp pinned by digest sha256:995fe9f2.... Serves qwen3.8-64k (alias) from ollama's GGUF blob, mounted read-only. Compose: compose/gpu-machine/llamacpp/. Replaced ollama as the server after a same-model/same-context benchmark: ollama 15.8 tok/s vs llama.cpp 39.7 tok/s (2.4x), because --cache-type-k/v q8_0 halves the KV cache (23.2GB -> 18.6GB) so the model fits the 3090 instead of spilling layers to CPU. Ollama has no KV-quantization knob. The image's own HEALTHCHECK probes :8080 and can never pass here, so it is disable: trued — liveness is an external GET of :8081/health.
    • friday-voice: :8320 — push-to-talk console. Calls speaches (host.docker.internal:8000) + hermes (docker exec hermes hermes -Q via wrapper). Public: coach.hobbs.farm → :8320. Compose: compose/gpu-machine/friday-voice/.
  • Ollama: TWO separate instances on this box, easy to confuse —
    • Native Windows ollama.exe v0.32.15, listens on ::11434 -> reachable as 192.168.86.150:11434. Holds qwen3.8, qwen3.8-64k, gemma4. This is the LAN-visible one and the source of the GGUF that llamacpp serves.
    • Inside WSL (/usr/local/bin/ollama serve, systemd unit, enabled+autostart) v0.15.5, relayed to Windows 127.0.0.1:11434 by wslrelay.exe. Holds qwen2.5:14b, hf.co/NousResearch/Hermes-4.3-36B-GGUF:Q4_K_M, qwen3.6, glm-4.7-flash, qwen3-coder:30b, gpt-oss:20b, llama3.1. host.docker.internal:11434 from a Docker Desktop container resolves to THIS one, not the native one — that cost real debugging time.
    • Both share the one physical 3090. Native ollama env (User scope): OLLAMA_KEEP_ALIVE=30s, OLLAMA_MAX_LOADED_MODELS=1, OLLAMA_NUM_PARALLEL=1 — keep-alive was deliberately dropped from -1 back to 30s once llamacpp became the server, so ollama stops pinning ~22GB of VRAM. These are read from the registry only by processes launched fresh (e.g. at login); a process started from an existing shell inherits that shell's env instead.
  • speaches (voicecoach-speaches-gpu) REMOVED 2026-08-21 to free ~3.3GB VRAM for the llama.cpp benchmark, per user ("they can be rebuilt later"). Image and the voicecoach-speaches_speaches-models volume were kept, so rebuild is just docker compose -f 'C:\Users\Jeremy\projects\docker\compose\gpu-machine\voicecoach-speaches\docker-compose.yml' up -d with no model re-download. FRIDAY's voice in/out (STT+TTS) is down until then; the text /api/ask path works. Its compose sets WHISPER__TTL=-1, which is why it pinned VRAM permanently — consider a finite TTL on rebuild.

Port Conflict Notes

Port conflicts already resolved:

  • Port 8000: Freed (MCPO moved to 8001, Portainer moved elsewhere)
  • Port 3000: freed 2026-07-13 (was AdGuard Home, decommissioned; Gitea remains on 3003)
  • Port 3001: Grafana (Semaphore moved to 3004)
  • Port 5000: Frigate uses host networking (registry container stopped)
  • AI Agent ports re-spaced (2026-03-15): openclaw 2026.3.13 uses gateway+2 and gateway+3 for browser/chrome-relay. Agents spaced 5 apart: margaret=18790, spirit=18795

Critical Configuration Preservation

Encryption Keys & Passwords

  • Semaphore: SEMAPHORE_ACCESS_KEY_ENCRYPTION encrypts sensitive data
  • Always check for existing encryption keys before generating new ones

Database Version Matching

Match the PostgreSQL/MySQL version to existing data directory format. Check logs for version mismatch errors.

Post-Change Checklist (MANDATORY)

After ANY infrastructure change (deploy, remove, reconfigure, add service, change ports, modify Traefik routes), you MUST complete ALL of these before reporting done:

  1. CLAUDE.md — Update service lists, ports, node assignments if changed
  2. MEMORY.md — Add/update relevant memory entries
  3. Prometheus probes — Add/remove blackbox targets, hot-reload Prometheus
  4. Wiki — Launch documentation-keeper agent to update wiki pages (run in background)
  5. Farm Manager links page (last audited 2026-10-26: every link probed; removed both FRIDAY Voice links, added Onyx (public + :3005), Guacamole :8199, Synapse Admin :8080, SearXNG :8888, n8n :5678, CouchDB Fauxton; MCPO -> /docs, Trivy -> /healthz (API only, no UI); Jellyseerr -> Seerr. Groups live in /mnt/docker-data/configs/farm-manager/groups.json -- update them too when containers are added/renamed. Previous audit 2026-08-20: dropped dead Clippy/Bob/Aaron links, added FRIDAY Voice + LinkedStorm + Rongorongo and a new GPU Machine (192.168.86.150) category; hermes intentionally absent — it publishes no ports. Served path is /app.js, not /static/app.js.) — If a service was added/removed or its URL/port changed: edit the serverLinks array in /opt/farm-manager/server/static/app.js on hf-pdocker-01, then rebuild + redeploy: docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d (links are baked into the image, not mounted)

Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.

Common Operations

Deploy/Update a Compose Service

  1. Edit compose/{node}/{service}/docker-compose.yml
  2. SSH to the target node and run: 
    cd /mnt/docker-data/compose/{node}/{service}
docker compose up -d

Troubleshoot

# On target node:
docker compose -f /mnt/docker-data/compose/{node}/{service}/docker-compose.yml logs --tail 100
docker ps --format "table {{.Names}}\t{{.Status}}"

Check Cluster Health

# On each node:
docker ps                         # Local containers

Data Management

Storage

All persistent data on NAS at /mnt/docker-data/:

  • configs/ - Application configurations
  • databases/ - Database volumes
  • media/ - Frigate recordings
  • traefik/ - SSL certificates

NAS Dependency

Risk: NAS failure = all services fail (single point of failure)

  • NFS mount uses soft,timeo=300 for graceful timeout
  • Synology should have "Restart automatically after power failure" enabled
  • Original data backups remain on local /data/configs/ on hf-pdocker-01

Specialized Agents

vault-credential-manager

Use for: Storing/retrieving credentials from Vaultwarden

documentation-keeper

Use for: Creating/updating documentation in Wiki.js

Known Issues & Workarounds

  1. NFS root_squash: Database containers need user: "1024:100" to avoid chown failures on NAS volumes

  2. Telegraf Docker Input: Disabled due to Docker socket permission issues across nodes

  3. Watchtower API Version: Requires DOCKER_API_VERSION=1.44

  4. InfluxDB Connection: Telegraf may show timeout errors if InfluxDB is not running

  5. Frigate cameras: 192.168.44.0/24 is the "Camera" VLAN 200 on the Firewalla (gateway 192.168.86.1 / 44.1). Cameras with 44.x IPs MUST sit on Camera-native switch ports. 2026-10-08: house_north/.51, house_east/.58, house_south/.53, house_west/.52 are wired to downstairsswitch (.159) ports 3/4/18/19 (verified from the switch MAC table 2026-10-11), which are VLAN 1, so they can't be reached by routing. They only half-worked because .192's Wi-Fi NIC held a wrong-VLAN 44.x lease. The fix is to make those ports Camera-native (see memory unifi-estate). The nodes' wlp2s0 Wi-Fi NICs must hold 86.x leases; a 44.x lease there is stale and breaks routing to the real Camera VLAN. 2026-10-10: with the Wi-Fi workaround gone (86.x lease since 10-08), those 4 cameras are now fully DOWN (0 fps, no ping) until the ports are re-profiled.

  6. Registry container: Stopped on hf-pdocker-01 to free port 5000 for Frigate

  7. systemd-resolved DNSStubListener: disabled on hf-pdocker-01 and hf-pdocker-02 via /etc/systemd/resolved.conf.d/nostub.conf (DNSStubListener=no) — originally for AdGuard Home's port 53. AdGuard was decommissioned 2026-07-13; the stub-listener change is harmless and left in place (re-enable only if something needs 127.0.0.53).

  8. NFS boot-mount race (hf-pdocker-01): NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s to both fstab entries. If mounts are missing after reboot: sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure.

  9. Guacamole schema: App image pinned to guacamole/guacamole:1.6.0 + guacd:1.6.0 with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.

  10. Gitea image: Pinned to gitea/gitea:1.26.1 with Watchtower disabled (2026-05-20). Watchtower previously pulled broken :latest images causing missing git binary errors.

  11. Home Assistant reverse proxy: Config at /mnt/docker-data/configs/homeassistant/configuration.yaml (no hyphen). Must have http: use_x_forwarded_for: true + trusted_proxies: [172.16.0.0/12, 127.0.0.1] or proxied requests return 400.

  12. hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11): Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: vm.swappiness lowered 60→10 (persisted in /etc/sysctl.d/99-swappiness.conf), and VoiceCoach (v1+v2, ~10 containers) removed (replaced by FRIDAY Voice on .150) — but see correction: the VoiceCoach compose stacks on .192 were only stopped in July, not down'd, and reappeared (~3GB) on the 2026-08-05 reboot; properly torn down 2026-08-05 (docker compose down on voicecoach + voicecoach-v2), node mem 9.0→6.7GB used. — this is WRONG, see 2026-08-20: only voicecoach-v2 is gone; the 6-container voicecoach project is still running on .192. Frigate is the main consumer. 2026-07-11: Frigate's built-in AI features (semantic_search: large, face_recognition, lpr, classification.bird) were found enabled (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config /mnt/docker-data/configs/frigate/config/config.yml (backup .bak.20260711-aitrim). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). 2026-07-12 — WHOLE-NODE CRASH: .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate memswap_limit had drifted to 16G vs mem_limit 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. Fix: node compose memswap_limit: 12G (= mem_limit) so Frigate can never use host swap — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep memswap_limit == mem_limit on this node.

  13. Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05): Both fight over \\.\pipe\docker_engine; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory docker-desktop-unifi-podman-conflict.

  14. MySQL InnoDB corruption on NFS: semaphore-db (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via --innodb-force-recovery=1 → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with semaphore user add. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory semaphore-db-nfs-corruption.

  15. litellm hangs after DB blip: litellm does NOT self-recover from a transient litellm-db disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual docker restart litellm. Its healthcheck was fixed from curl (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).

  16. Stale NFS locks after unclean reboot/power loss (2026-07-12 power event): NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on ibdata1, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → /var/lib/gitea-queues, semaphore-db → /var/lib/semaphore-db on .100; sonarr config → /var/lib/sonarr, nzbhydra config → /var/lib/nzbhydra on bart; jellyfin config → /var/lib/jellyfin on bart 2026-07-19 after 60-100s "database is locked" stalls on jellyfin.db caused playback lockups; synapse data → /var/lib/synapse on .192 2026-08-20, pre-emptively — its 279MB SQLite homeserver.db was on NFS). Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.

  17. Synapse rate-limited Alertmanager notifications: default rc_message (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — no alerts reached Matrix for weeks while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: rc_message: {per_second: 5, burst_count: 50} in homeserver.yaml (backup .bak-20260713-ratelimit).

  18. AdGuard Home decommissioned 2026-07-13: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router.

  19. Duplicate failover stacks removed from .100 and bart (2026-07-13): .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.

  20. UniFi OS Server probe port (historical): the unifi-os-server container on .100 served HTTPS on host port 11443 (container 443), not 8443. That container was removed 2026-10-05 (duplicate controller) — the sole UniFi controller is now the Podman instance on .150 (UI https://192.168.86.150:11443, inform :8080). The live Prometheus config did carry a blackbox probe for https://192.168.86.100:11443 (line 132) — it was removed 2026-10-06 (backup prometheus.yml.bak-20261006-unifi-probe-remove) and Prometheus hot-reloaded, since nothing listens there now. No probe added for the .150 controller because that box is a workstation that is sometimes off (it would false-alarm); the switch itself (192.168.86.159) could be ICMP-probed instead if device-down alerting is wanted. (Note: the UnifiDeviceDown alert in alerts.yml keys on unifi_device_up, which needs a unifi-exporter that is NOT deployed — the metric doesn't exist, so that rule is inert, not firing.)

  21. OpenVAS pg_upgrade loop saturated bart's HDD (2026-07-15): immauss/openvas:latest looped on the PG 13→15 auto-upgrade for ~21h (158 restarts) — each cycle re-copied the 21GB DB, pinning sda at ~90% util/40% iowait and making Jellyfin (same disk) hang. The upgrade itself succeeded every time but the final mv /data/database /data/database_13_old failed with "Device or resource busy" because the openvas-db volume is mounted at /data/database (can't rename a mountpoint). Fixed by wiping the DB (docker volume rm openvas_openvas-db) and letting it initdb fresh on PG15 — scan history lost, feeds re-synced. On the next major PG bump this will recur unless the upgrade is done out-of-band (content-level swap) or the volume is remounted at /data. General rule: a service "hanging" on bart usually means another container is eating the single HDD — check iostat -x/pidstat -d first. OpenVAS fully decommissioned 2026-07-19 (moot unless redeployed).

  22. Jellyfin SQLite-on-NFS lockups (fixed 2026-07-19): jellyfin.db (WAL mode) on /mnt/docker-data/configs/jellyfin hit escalating SQLite Error 5: 'database is locked' (22→33→125/day), DB commands blocking 60-102s → healthcheck flaps + playback stalls ("Unexpected end of request content" on /Sessions/Playing/Progress). Config moved to LOCAL /var/lib/jellyfin (old copy left on NFS as backup). Same class as Known Issue #16.

  23. Matrix registration was wide open to the internet (fixed 2026-08-20): mess.hobbs.farm is publicly reachable (Traefik, port 443, federation + client API on :8008) and homeserver.yaml had enable_registration: true + enable_registration_without_verification: true — the registration flow was a bare m.login.dummy, i.e. anyone on the internet could create an account on hobbs.farm with no email, captcha, or token. No abuse had occurred (all 18 accounts are known/expected; newest legit account was @fry 2026-07-05). Fixed by setting both to false and restrict_public_rooms_to_local_users: true (was false, which published the public room directory to any remote server). Backup: /var/lib/synapse/homeserver.yaml.bak-20260820-registration. New accounts are still creatable on the box via docker exec synapse register_new_matrix_user -k <registration_shared_secret>.

  24. Synapse database history (SQLite -> local disk -> PostgreSQL, all 2026-08-20): Synapse ran on SQLite on NFS while CLAUDE.md claimed it used a synapse-db postgres that in fact had 0 tables. First the data was moved to LOCAL /var/lib/synapse and the empty postgres deleted; then, because syn2mas only reads PostgreSQL ("The migration tool only supports reading from PostgreSQL for the Synapse database as well"), a real synapse-db was created and the DB ported with synapse_port_db (20 users / 93,888 events / 65 devices, verified identical; 2m35s downtime). The PG database MUST have LC_COLLATE=C -- verify with select datcollate from pg_database where datname='synapse' (PG16 removed the SHOW lc_collate GUC). The old SQLite file is still at /var/lib/synapse/homeserver.db, now inert; NFS copy at /mnt/docker-data/configs/matrix/data is an older backup -- never remount either.

  25. Synapse upgraded 1.148.0 -> 1.159.0 and moved to the element-hq registry (2026-08-20): it had been on matrixdotorg/synapse:latest, an abandoned Docker Hub repo whose :latest froze at 1.148.0 (image built 2026-02-24) — a moving tag that silently stopped moving on a publicly-exposed homeserver. Now pinned to ghcr.io/element-hq/synapse:v1.159.0 (note the v prefix — :1.159.0 without it 404s). Upgrade was clean: schema 93 -> 94, all background updates finished in ~5s, 0 errors, 19 users / 93,884 events intact, memory 112MB/1G. None of the v1.149-v1.159 breaking changes applied to us (MSC3861 removal — unused; quarantined_media_changes stream writer — workers only; the rest are Debian/Ubuntu packaging). Pre-upgrade backup: /var/lib/synapse.bak-20260820-pre1159 (646MB) — delete once you're satisfied. Pinned on purpose: bump it manually and deliberately, since the last moving tag rotted for 6 months unnoticed. 2026-10-26: bumped v1.159.0 -> v1.163.2 (schema 94 deltas applied in <1s; config contains none of the options affected by the 1.160-1.163 upgrade notes). pg_dump backups taken first: /var/lib/upgrade-backups-20261026/{synapse-db,mas-db}.dump on .192. MAS bumped 1.23.0 -> 1.27.1 at the same time.

  26. SYNAPSE_SERVER_NAME in the matrix compose disagrees with homeserver.yaml: compose passes SYNAPSE_SERVER_NAME=mess.hobbs.farm but the real server_name is hobbs.farm (user IDs are @user:hobbs.farm). The env var is only read when the image generates a config, so it's inert today — but if homeserver.yaml ever went missing, the container would generate a fresh config with the wrong server_name AND a new signing key, which would break federation and every existing user ID. Left as-is (changing it has no effect while the config exists); just never let the container auto-generate..

  27. Matrix auth is delegated to MAS -- Synapse does NOT serve /login any more (2026-08-20): with matrix_authentication_service.enabled: true, Synapse answers M_UNRECOGNIZED on /_matrix/client/*/login. The reverse proxy must route the three compat paths to MAS or nobody can sign in: login, logout, refresh (verified: MAS returns 200/405 on those and 404 on register/whoami, which stay on Synapse). Implemented as the mess-auth-compat router in /mnt/docker-data/traefik/dynamic/temp-routes.yml with explicit priority: 1000 so it beats the catch-all mess router. Anything that talks to 192.168.86.192:8008 directly bypasses Traefik and therefore cannot log in with a password -- token-based bots (spirit, frigate, alertbot) are unaffected because they never call /login, but @clippyauto (n8n) and @pluto historically re-login with a password (17 and 15 accumulated devices). They currently run on migrated tokens; if either ever needs a fresh password login, point it at https://mess.hobbs.farm instead of :8008.

  28. MAS image tags are NOT v-prefixed -- ghcr.io/element-hq/matrix-authentication-service:1.23.0 is correct, :v1.23.0 does not exist. This is the opposite of Synapse (ghcr.io/element-hq/synapse:v1.159.0, where :1.159.0 404s). The MAS image is also distroless -- no shell, so a container healthcheck is impossible; liveness comes from the blackbox probe of http://192.168.86.192:8021/health. Note /health lives on the internal listener only: https://auth.hobbs.farm/health returns 404 by design, so the public probe targets /.well-known/openid-configuration.

  29. MAS config holds unrecoverable signing keys: /mnt/docker-data/configs/mas/config.yaml (mode 640) contains the RSA/EC signing keys and the matrix.secret shared with Synapse. Regenerating them invalidates every session. It is not yet backed up to Vaultwarden because the vault master password is currently unknown (see memory vault-master-password-stale) -- resolve that and store it. Also note http.issuer/public_base (https://auth.hobbs.farm/) are baked into session state; changing the hostname later invalidates sessions.

  30. OpenClaw agent images were built from openclaw@latest and rotted for months (fixed 2026-08-20): margaret and spirit sat on 2026.5.27 while upstream was on 2026.7.1-2 — same failure mode as the abandoned matrixdotorg/synapse:latest (Known Issue #25). Worse, the Dockerfiles existed nowhere — not in the repo, not on any node — and had to be reconstructed from docker history layer output. They now live at compose/hf-pdocker-01/ai-agents/{margaret,spirit}/Dockerfile (staged on the NAS at /mnt/docker-data/compose/hf-pdocker-01/ai-agents/) with ARG OPENCLAW_VERSION pinned; bump it deliberately. Rebuild: cd /mnt/docker-data/compose/hf-pdocker-01/ai-agents && docker build -t margaret-openclaw:<ver> -t margaret-openclaw:latest ./margaret (same for ./spirit → spirit-ops). npm dist-tags to check: latest, extended-stable, beta. Upgrading migrates the state dir one-way (JSON sidecars → shared SQLite; originals archived as *.migrated under /mnt/infrastructure/{margaret,spirit}/), so a rollback to *:rollback-2026.5.27 also needs those .migrated files restored. Config backups: /mnt/infrastructure/{margaret,spirit}/openclaw.json.bak-20260820-preupgrade.

  31. 192.168.86.27 is dead and several configs still point at it (2026-08-20): spirit's primary model was vllm-local/qwen at 192.168.86.27:8001, so every message burned ~12s timing out before failing over to Anthropic (2,913 consecutive primary-model failures). margaret had the same host as a vllm-local/hermes provider. Both vllm-local provider blocks were deleted; both agents are now anthropic/claude-sonnet-4-6 primary with google/gemini-2.5-flash fallback. voicecoach-openclaw still points at 192.168.86.27:11434 with no fallbacks. There is no host at .27 — grep any new agent config for it. Related: margaret listed google/gemini-2.5-flash as a fallback while defining no google provider or auth profile, so it could never have fired; a google provider block (api google-generative-ai, key verified against the live models list) was added to match spirit's working shape.

  32. CORRECTED 2026-08-21 — there are TWO ollama instances on .150, which is why the model lists disagreed: a native Windows ollama.exe (v0.32.15) on 192.168.86.150:11434 holding only gemma4 + the new qwen3.8*, and one inside WSL (v0.15.5, systemd, autostart) relayed to 127.0.0.1:11434, holding qwen2.5:14b, Hermes-4.3-36B-GGUF, qwen3.6, qwen3-coder:30b, gpt-oss:20b, llama3.1. So the older docs were right — those models exist, just in the WSL instance. An earlier note here claiming "only gemma4" was measuring the native one alone. host.docker.internal:11434 from a Docker Desktop container hits the WSL instance; the LAN IP hits the native one. Both share the single 3090, so either can starve the other. spirit's ollama provider still advertises a nonexistent qwen3.5 (harmless, not in its chain). LinkedStorm on .192 points at 192.168.86.150:11434 for qwen2.5:14b — that model is NOT on the native instance, so LinkedStorm's LLM calls are probably 404ing. Still unverified.

  33. Docker Desktop on .150 died twice while loading a large model (2026-08-21): the WSL2 VM had no .wslconfig, so it defaulted to 50% of host RAM = 15GB shared by the Ubuntu distro, the docker-desktop VM and podman-uosserver. Loading llama.cpp's 16.8GB GGUF alongside the running containers OOM-killed hermes and friday-voice (exit 137), and on the first attempt took the whole Docker Desktop backend down — it went unable to start -> HTTP 500 on every API call -> engine pipe missing entirely. Recovery that worked: kill all Docker Desktop/com.docker processes, wsl --terminate docker-desktop, then start Docker Desktop before UniFi OS Server (the pipe-conflict rule in issue #13 still applies — UniFi went down in the crash and had to be restarted after). Mitigation applied: C:\Users\Jeremy\.wslconfig now sets memory=22GB, swap=8GB — this needs wsl --shutdown or a reboot to take effect, and wsl --shutdown kills the Claude Code session too, so it has NOT been applied yet. Until it is, expect OOM churn when a big model loads.

  34. Onyx self-registration was open to the internet (fixed 2026-09-21): onyx.hobbs.farm is publicly reachable and Onyx runs AUTH_TYPE=basic with no REQUIRE_EMAIL_VERIFICATION and no VALID_EMAIL_DOMAINS — so POST /api/auth/register let anyone on the internet create an account. External scanners were already probing the host (a DigitalOcean IP hit / on 2026-09-21). No abuse had occurred: only 2 accounts existed (glyptic@gmail.com + Onyx's internal anonymous@onyx.app). Fix: a Traefik router onyx-register-lockdown (priority: 1000, beats the catch-all onyx router) matching PathPrefix(/api/auth/register) behind an ipAllowList middleware limited to 192.168.86.0/24 + 127.0.0.1/32. Verified by temporarily narrowing the range: register returned 403 from a non-allowlisted source while /api/auth/login (405) and / (200) were unaffected; range then restored. Config /mnt/docker-data/traefik/dynamic/onyx.yml (backup .bak-20260920-195851). To add a user: register from the LAN, or widen the sourceRange. Note VALID_EMAIL_DOMAINS was rejected as the control because it is domain-level only — gmail.com would admit any Gmail user. Same class as Known Issue #23 (open Matrix registration).

  35. The live Prometheus config is /mnt/docker-data/configs/prometheus/prometheus.yml — the copy at compose/hf-pdocker-01/monitoring/prometheus.yml is STALE and misleading (2026-09-21): the compose-dir copy still lists decommissioned targets, including Onyx at the wrong node (192.168.86.167:3005, bart) and OpenVAS (:9392). Neither is in the live config, so neither was ever probed — grepping the compose copy to answer "is X monitored?" gives the wrong answer. Onyx had no probe at all until one was added to the live file on 2026-09-21 (http://192.168.86.192:3005, now probe_success=1). Backup prometheus.yml.bak-20260920-*. The live file is also the better changelog: it records the voicecoach (2026-08-24) and FRIDAY Voice / speaches (2026-08-26) teardowns that CLAUDE.md still described as live.

  36. SSH key auth to hf-pdocker-01 is broken — password only (2026-09-21): all of ~/.ssh/id_docker_server, id_bart, id_nas are rejected by 192.168.86.192 (Permission denied (publickey,password)), while id_docker_server still works for .100 and id_bart for bart. Connect with sshpass -p '&Rip8a115' ssh -o PubkeyAuthentication=no glyptic@192.168.86.192. Any key-based automation targeting .192 is silently failing — check ~/.ssh/authorized_keys on that node (likely lost when the host was rebuilt/restored).

  37. hf-pdocker-01 hard-hung for ~27h (2026-10-09 13:55 → 2026-10-10 17:08 CDT): journal simply stops at 13:55 with no OOM, kernel error, or shutdown record; sar at 13:50 showed a healthy box (62% mem, 23% swap, load ~4), and .100/bart stayed up (9d uptime), so it was neither memory exhaustion nor a house-wide power event. Came back on kernel 7.0.0-38 (was -34) after a power cycle. Cause unknown — if it recurs, suspect hardware (this box was repaired 2026-03) and consider enabling persistent kdump/netconsole. Side effects found on 2026-10-10: onyx-opensearch-1 revived itself (fixed, see Onyx entry). Also fixed then: ServiceHighMemory used container_memory_usage_bytes, which counts reclaimable page cache, so clip-service false-alarmed at 94% while really at 64% — rule now uses container_memory_working_set_bytes (backup alerts.yml.bak-20261010-workingset).

  38. bart shows ~20 nameless Dead containers in docker ps -a — orphaned /var/lib/docker/containers/<id> dirs for containers whose images Watchtower already deleted. docker rm says "No such container". Harmless clutter; clearing them requires stopping dockerd (moving the dirs aside) which restarts the whole media stack, so it was left for a maintenance window.

  39. [RESOLVED 2026-10-11] Static-IP ESPHome boards were invisible behind UniFi Proxy ARP (found 2026-10-10): the ASUS_F0_2G WLAN has Proxy ARP on (proxy_arp=1 in every AP's /etc/hostapd/wifi0ap0.cfg; it also has ap_isolate=1 and mcast.enhance=1). With Proxy ARP, the AP answers ARP for wireless clients from a table it learns by snooping DHCP and stops passing ARP broadcasts through to them. Boards with ESPHome manual_ip never DHCP, so after any reconnect nobody answers ARP for them: they associate fine and even announce themselves over mDNS, but are unreachable. Pool Value Controls (.6, MAC a4:cf:12:f1:6e:c8) went dark 2026-10-09 11:04 at a Wi-Fi reconnect on AP .187 and then looped every 15 min on ESPHome's API reboot_timeout. Diagnosed by capture on .187: ARP who-has .6 reached the radio and got no reply, while a static neighbor entry made ping and API work at once. Workaround (live): systemd unit static-arp-esphome.service on .192 pins .6 -> a4:cf:12:f1:6e:c8 nud permanent on enp0s31f6 (HA and ESPHome are bridge-networked on .192, so they use the host's neighbor table). Real fix (pending, needs controller access on .150): turn Proxy ARP off on the ASUS_F0_2G WLAN, OR switch the boards to DHCP + Firewalla reservations. Once Proxy ARP is off, systemctl disable --now static-arp-esphome on .192. A power_save_mode: none change was also OTA'd to the board (backup pool-value-controls.yaml.bak-20261010-powersave). It did NOT fix this, but it's harmless and was left in. Any other static-IP Wi-Fi device on that SSID has the same problem. RESOLVED 2026-10-11: Proxy ARP (and ap_isolate) were turned off on ASUS_F0_2G in the controller (pushed to all APs 11:09–11:18; the proxy_arp/ap_isolate lines are gone from every AP's hostapd config). static-arp-esphome.service was then disabled and deleted on .192. Verified afterwards: .6 resolves by normal ARP (neighbor REACHABLE, not PERMANENT), ping OK, HA API session established. If a static-IP Wi-Fi device goes dark again, check that Proxy ARP hasn't been switched back on. polelightpond (.14) and living_room_lights (.153) were still unreachable after the fix, so they have some other problem (unpowered or offline).

  40. Container version audit + upgrade round (2026-10-26). Upgraded: Synapse v1.163.2, MAS 1.27.1, Traefik v3.7 (3.7.16), node-exporter v1.12.1 (all 3 nodes), cadvisor v0.60.6, matrix-alertmanager-receiver 2026.9.23, Gitea 1.26.4 (patch only; pg_dump at ~/gitea-db-20261026-pre1264.dump on .100), Jellyfin 12.3, Seerr v3.5.1, plus re-pulls of every moving tag on .192 (HA 2026.10.4, ESPHome, Mosquitto, Prometheus, Blackbox, Telegraf, nginx, litellm, and the postgres/mysql/redis patch images). Every compose file touched has a .bak-20261026 next to it on the NAS. Deliberately NOT done, each needs its own decision: Onyx :latest now = v5.0.0 (major, released 2026-10-24), still on v4.9.2 (no Watchtower on .192, so it only moves on a manual pull); Gitea 1.27.x / 28.x (major jumps); OpenClaw 2026.7.1-2 -> 2026.10.7 (one-way state migration, see #30); hermes v2026.8.18 -> v0.22.5 (upstream switched to semver tags, and its LLM endpoint is dead anyway); MySQL 8.0 (guacamole-db, semaphore-db) went EOL April 2026 -> needs an 8.4 LTS migration with a dump/restore; InfluxDB 1.8 (EOL); Onyx's postgres:15.2-alpine (2023 patch level, managed by Onyx's compose); awesometechnologies/synapse-admin (stale, maintained fork is etkecc/synapse-admin). How to audit: compare each container's RepoDigests against registry HEAD digests -- for Docker Hub use the hub.docker.com web API (/v2/namespaces/<ns>/repositories/<repo>/tags/<tag> -> digest), which does not count against pull limits. docker buildx imagetools inspect --format is ignored by the nodes' buildx v0.31.1.

  41. Docker Hub anonymous pull limit is shared by the whole house (public IP 98.160.103.126, 100 pulls/hour): on 2026-10-26 every node AND WSL got 429 toomanyrequests, meaning Watchtower updates of Docker Hub images were silently failing. Mitigation applied: both Watchtowers moved from a 24h poll interval to a twice-daily cron schedule (04:00/16:00), and the archived containrrr/watchtower swapped for the maintained drop-in fork nickfedor/watchtower (image name is nickfedor/, not nicholas-fedor/ -- the latter is the GitHub org and 404s on Hub). Same env vars and still honors com.centurylinklabs.watchtower.enable. A docker login on the nodes would raise the limit further. cadvisor no longer publishes to gcr.io after v0.55 -- the image is now ghcr.io/google/cadvisor.

  42. node-exporter had been launched from /tmp (compose project tmp) on all 3 nodes, so docker compose in the real dirs didn't own it. Fixed 2026-10-26: removed and redeployed from /mnt/docker-data/compose/<node>/node-exporter/. Separately, the monitoring compose on .192 still defines a legacy node-exporter:latest service on :9100 that a bare docker compose up -d would try to start; it is now gated behind profiles: ["legacy-node-exporter"]. Also: the matrix-alertmanager-receiver container is now named monitoring-matrix-alertmanager-receiver-1 (compose default) -- Alertmanager reaches it by service name matrix-alertmanager-receiver, which still resolves.

  43. Farm Manager showed ZERO containers for hf-pdocker-01 (fixed 2026-10-26): the server fetches each agent's /containers with a timeout and on any error silently returns [] for that node. .192's agent took ~12s (47 containers, per-container stats(stream=False) = Docker samples twice ~1s apart, only 10 workers) vs a 10s timeout, so the main host's containers simply vanished from the dashboard while /api/nodes still said "healthy, 47". Fix: agent uses stats(..., one_shot=True) + 20 workers (now ~2s), server timeout 30s. Agent rebuilt on .192 and shipped to .100/bart via docker save | docker load (only .192 has the /opt/farm-manager source); rollback tags farm-agent:rollback-20261026 / farm-manager:rollback-20261026. Same day: removed a dead gpu-workstation node (192.168.86.27) from the live config.json, made /health ignore Dead containers (bart reported 30 instead of 10), and rebuilt groups.json -- every group still used Swarm-era / decommissioned names (ai-agents_margaret, jellyseerr, adguardhome, bart Onyx, openvas, unifi...), so all group actions were no-ops. Now 14 groups / 63 containers; per-node utilities (farm-agent, node-exporter, telegraf, watchtower) are deliberately ungrouped so a group "stop" can't kill the agents. 4 tests in tests/test_agent_docker_ops.py fail -- pre-existing (they mock containers.list but the code uses low-level api.containers); not caused by this change.

Reference in New Issue View Git Blame Copy Permalink