jeremy/hobbs-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: UniFi controller consolidation, STP fix, NFS recovery

Browse Source 
- Remove duplicate unifi-os-server container from .100; sole UniFi
  controller is now the .150 Podman instance (UI :11443, inform :8080).
  Data preserved; compose dir kept.
- Document STP root-cause/fix for wifi client drops: version mismatch
  (stp vs rstp) + default priorities -> root flapping; fixed by RSTP
  everywhere + .159 pinned root @4096.
- Known Issue #20: correct the stale .100:11443 blackbox probe note -
  the probe DID exist in the live config and was removed 2026-10-06
  (backup saved) + Prometheus reloaded; UnifiDeviceDown rule is inert.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
jeremy
2026-10-06 19:06:05 -05:00
parent 9a81138cc9
commit beb727528a
1 changed files with 36 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+36 -16
View File
@@ -23,14 +23,14 @@ This repository manages a **homelab infrastructure** with self-hosted services r
hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
├── [REMOVED] clip-service -- decommissioned 2026-07-13 (CLIP embeddings; ~2GB RAM freed, no config references from frigate-notify existed)
├── [COMPOSE] clip-service (:5200) -- **RUNNING, and actively used.** CORRECTED 2026-09-21: the 2026-07-13 decommission note below was wrong for the current deployment — it was redeployed and the *live* `frigate-notify/docker-compose.yml` sets `CLIP_SERVICE_URL=http://localhost:5200`, with live `app/recognition.py` calling `/embed`. ~1.2GB RSS. **Do not remove without first rewiring frigate-notify's recognition pipeline.**
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] ESPHome (:6052)
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Synapse/Matrix (:8008) + wellknown (:8010) + synapse-db (**postgres:16-alpine, LOCAL /var/lib/synapse-db, C collation**) -- Synapse data on LOCAL /var/lib/synapse. Migrated SQLite->PostgreSQL 2026-08-20 (required by syn2mas). **Auth is delegated to MAS** -- Synapse no longer serves /login.
├── [COMPOSE] **MAS / Matrix Authentication Service (:8020 web, :8021 health) + mas-db (postgres:16-alpine, LOCAL /var/lib/mas-db)** -- OAuth2/OIDC provider for Matrix. Public: auth.hobbs.farm. Deployed 2026-08-20 to enable QR sign-in (MSC4108). QR sign-in **verified working end-to-end 2026-08-20**. **Every Matrix login now depends on this service.**
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops]) -- **OpenClaw 2026.7.1-2, images PINNED via Dockerfile ARG** (`compose/hf-pdocker-01/ai-agents/{margaret,spirit}/Dockerfile`). Both primary `anthropic/claude-sonnet-4-6`, fallback `google/gemini-2.5-flash`.
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192.
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
@@ -39,6 +39,8 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
├── [COMPOSE] **Onyx (RAG/search) -- 10 containers, `/mnt/docker-data/compose/hf-pdocker-01/onyx`.** NOT decommissioned (the bart entry below refers to an older, separate deployment). Public: **onyx.hobbs.farm** -> nginx :3005. Custom build: `gitea_patch.py` + `unstructured_patch.py` index Gitea (`git.hobbs.farm/api/v1`) and call unstructured-api on .100:8000. LLM -> `192.168.86.150:18020/v1` (`qwen3.8-27b`) — **dead while .150 is down.** `onyx-opensearch-1` **stopped 2026-09-21** (`ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false`, held no Onyx data, nothing `depends_on` it; freed ~1.1GB). Restart with `docker compose up -d opensearch` if the flag is ever flipped.
├── [COMPOSE] hermes -- **moved here from .150** (host network, `/mnt/docker-data/compose/hf-pdocker-01/hermes`, pinned `v2026.8.18`). Points at `http://192.168.86.150:18020/v1` model `qwen3.8-27b`; that endpoint is **dead**, so every call burns 200s x3 retries and fails. Note this is a different host:port than the `.150:8081` llamacpp setup documented in the GPU Machine section.
├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
├── [COMPOSE] synapse-admin -- Synapse admin UI
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
@@ -48,7 +50,7 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12.
├── [REMOVED] Onyx -- fully decommissioned 2026-07-13 (had been down since ~2026-06-08 unnoticed; containers, volumes, images removed)
├── [REMOVED] Onyx -- this *bart* deployment was decommissioned 2026-07-13. **A different, newer Onyx now runs on hf-pdocker-01 (see above) — 'Onyx is gone' is false.**
├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15.
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
@@ -56,7 +58,7 @@ bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Watchtower
hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
├── [COMPOSE] UniFi OS Server (container `unifi-os-server`, HTTPS on host :11443)
├── [REMOVED] UniFi OS Server -- the `unifi-os-server` container (ghcr.io/lemker/unifi-os-server, v5.1.15) was decommissioned 2026-10-05. It was a **duplicate/idle** controller -- the real UniFi controller is the **Podman UniFi OS Server on the Windows box (.150)**, which manages all UniFi devices (e.g. `downstairsswitch` US-24-250W at 192.168.86.159, inform `http://192.168.86.150:8080/inform`). Container + image + network removed; ports 11443/8080/3478/etc freed on .100. **Data preserved** at `/var/lib/unifi-os/*` (971M local) + `/mnt/docker-data/configs/unifi-os/persistent` (NAS); compose dir kept for reference. **Only .150 manages UniFi, by design.**
├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
@@ -146,8 +148,9 @@ compose/ # Standalone Compose definitions (deployed per-node)
- Home Assistant: `yoda.hobbs.farm`
- Vaultwarden: `vault.hobbs.farm`
- Gitea: `git.hobbs.farm`
- FRIDAY Voice: `coach.hobbs.farm` (replaced VoiceCoach 2026-07-05; runs on .150 :8320) — **DOWN (502) as of 2026-08-05, intentionally left down.** .150 Docker Desktop is offline so FRIDAY isn't running; a 2026-08-02 stopgap had repointed the Traefik route to a .192 voicecoach-web failover (`:8085`), which was removed in the 2026-08-05 VoiceCoach teardown → route now points at a dead backend. To restore: start Docker Desktop on .150, up friday-voice, repoint `friday-voice.yml` back to `.150:8320`. See memory `friday-voice`.
- FRIDAY Voice: `coach.hobbs.farm` — **DECOMMISSIONED 2026-08-26; `coach.hobbs.farm` returns 404, which is EXPECTED, not an outage.** CORRECTED 2026-09-21: the "UP as of 2026-08-20" text previously here was overtaken by events. The live Prometheus config records the `voicecoach` teardown (2026-08-24) and the FRIDAY Voice + speaches teardown (2026-08-26) — routing and monitoring sides done, with "container/volume teardown on .150 still pending manual action next Windows boot". The 6-container `voicecoach` project is indeed gone from .192 (verified 2026-09-21). Its `.150` images/volumes may still exist. See memory `friday-voice`.
- MAS (Matrix auth): `auth.hobbs.farm` -- OAuth2/OIDC issuer. **If this is down, nobody can log into Matrix** (existing tokens keep working).
- Onyx (RAG/search): `onyx.hobbs.farm` -- **self-registration is LAN-only** (Traefik `onyx-register-lockdown` router). See Known Issue #34.
- Rongorongo: `rongorongo.hobbs.farm`
- LinkedStorm: `linkedstorm.hobbs.farm`
@@ -170,16 +173,18 @@ compose/ # Standalone Compose definitions (deployed per-node)
- InfluxDB: :8086
- Blackbox Exporter: :9115
- matrix-alertmanager-receiver: :9094
- AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0)
- AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0). OpenClaw **2026.7.1-2** (upgraded 2026-08-20 from 2026.5.27). Matrix: `@margaret:hobbs.farm` / `@spirit:hobbs.farm`, token auth against `http://192.168.86.192:8008`, encryption off, `requireMention: false`. Rollback images kept as `margaret-openclaw:rollback-2026.5.27` / `spirit-ops:rollback-2026.5.27`.
- Onyx: :3005 (onyx-nginx). Public onyx.hobbs.farm. `onyx-opensearch-1` deliberately stopped 2026-09-21.
- hermes: no published port (host network) -- see service tree; its LLM endpoint is currently dead.
- Farm Manager: :8888 (dashboard + API)
- Farm Agent: :8889
- Rongorongo: :8181 (rongorongo-web nginx; reverse-proxies /api/* to rongorongo-api Flask backend on the internal `rongorongo` bridge net). State + images at /mnt/docker-data/configs/rongorongo/data/. Public URL gated by Traefik basicAuth (creds in Vaultwarden).
- LinkedStorm: :8300 (pipeline/VAD+STT+speaker-ID), :8310 (API/dashboard), cron (no port). Public: linkedstorm.hobbs.farm -> :8310. Source: /opt/linkedstorm (rsync from WSL). Data: /mnt/docker-data/configs/linkedstorm/. STT via Speaches GPU machine :8000, LLM via Ollama :11434 (qwen2.5:14b).
- ~~VoiceCoach~~ -- replaced by FRIDAY Voice on .150 (see GPU Machine section + memory `friday-voice`). **NOTE: the on-.192 stacks were only *stopped* in July, never torn down — the 2026-08-05 power-cycle reboot brought all 7 containers (~3GB RAM, incl. speechbrain 1.6GB) back. Actually `docker compose down`'d 2026-08-05** on both compose projects `voicecoach` + `voicecoach-v2` (dirs `/mnt/docker-data/compose/hf-pdocker-01/voicecoach{,-v2}` kept for reference). Containers + networks removed; won't restart on reboot now. Do NOT `docker compose up` those dirs.
- ~~VoiceCoach~~ -- replaced by FRIDAY Voice on .150 (see GPU Machine section + memory `friday-voice`). **CORRECTION 2026-08-20: the teardown claimed below never actually happened.** All 6 containers of compose project `voicecoach` (`voicecoach-web` :8085, `-pipecat`, `-speaches`, `-speechbrain`, `-openclaw` :3100, `-livekit` :7880-7881 + 50000-50100/udp) are **running on .192, up since ~2026-08-06**. Earlier text said both projects were `docker compose down`'d on 2026-08-05; that is false for `voicecoach` (`voicecoach-v2` is genuinely gone). ~1GB RAM. `voicecoach-openclaw` was upgraded to OpenClaw 2026.7.1-2 on 2026-08-20 via `docker compose up -d --no-deps voicecoach-openclaw` (single service only — does not resurrect anything, the rest were already up). Its model is `ollama-local/qwen2.5:14b` at the **dead** `192.168.86.27:11434` with **no fallbacks**, so it cannot serve a turn. **Decide: fix its model chain or actually tear the stack down.**
**hf-pdocker-02 (192.168.86.100):**
- Farm Agent: :8889
- UniFi OS Server: :11443 (HTTPS, container 443)
- ~~UniFi OS Server: :11443~~ -- **REMOVED 2026-10-05** (duplicate controller; see service tree). The UniFi controller lives on **.150 (Podman), UI at https://192.168.86.150:11443, inform :8080**.
- Gitea: :3003 (web), :2222 (SSH)
- Semaphore: :3004
- Wiki.js: :3002
@@ -198,13 +203,18 @@ compose/ # Standalone Compose definitions (deployed per-node)
- Jellyseerr: :5055
**GPU Machine (192.168.86.150, RTX 3090):**
- **NOTE (2026-07-05):** .150 is actually the user's **Windows workstation** (runs the WSL/Claude Code session), not a headless Linux server. It runs **Docker Desktop** (WSL2 backend, RTX 3090 GPU passthrough working) alongside **UniFi OS Server** (Podman). Docker Desktop panics if UniFi/Podman claims `\\.\pipe\docker_engine` first — start Docker Desktop before UniFi OS Server. See memory `docker-desktop-unifi-podman-conflict`.
- **NOTE (2026-07-05):** .150 is actually the user's **Windows workstation** (runs the WSL/Claude Code session), not a headless Linux server. It runs **Docker Desktop** (WSL2 backend, RTX 3090 GPU passthrough working) alongside **UniFi OS Server** (Podman) — **this is the ONE real UniFi controller** (the duplicate container on .100 was removed 2026-10-05). UI https://192.168.86.150:11443, inform http://192.168.86.150:8080. Manages all UniFi devices incl. `downstairsswitch` (US-24-250W, 192.168.86.159; SSH `admin`/node-password). Docker Desktop panics if UniFi/Podman claims `\\.\pipe\docker_engine` first — start Docker Desktop before UniFi OS Server. See memory `docker-desktop-unifi-podman-conflict`.
- **FRIDAY Voice replaced VoiceCoach (2026-07-05).** VoiceCoach fully decommissioned (v1+v2 removed from hf-pdocker-01; voice.hobbs.farm route removed). See memory `friday-voice`.
- [COMPOSE] **FRIDAY Voice stack (all on .150 Docker Desktop, via `docker.exe -c desktop-linux`):**
- **voicecoach-speaches-gpu**: :8000 — GPU Whisper large-v3-turbo (STT) + Kokoro-82M af_nova (TTS), CUDA float16. Compose: `compose/gpu-machine/voicecoach-speaches/`.
- **hermes**: `nousresearch/hermes-agent:latest` gateway — FRIDAY's reasoning agent. Settings transferred from WSL `~/.hermes` into named volume `hermes-data`. Compose: `compose/gpu-machine/hermes/`.
- **voicecoach-speaches-gpu**: :8000 — GPU Whisper large-v3-turbo (STT) + Kokoro-82M af_nova (TTS), CUDA float16. Compose: `compose/gpu-machine/voicecoach-speaches/`. **REMOVED 2026-08-21 (see the speaches note at the end of this section) — rebuild when voice I/O is wanted again.**
- **hermes**: **PINNED `nousresearch/hermes-agent:v2026.8.18`** (upgraded 2026-08-20 from v2026.7.1 / v0.18.0 → v0.20.4). FRIDAY's reasoning agent. Settings in named volume `hermes-data`. Compose: `compose/gpu-machine/hermes/`. Was on `:latest` **with Watchtower disabled**, so nothing ever re-pulled it — it sat 7 weeks / 9 releases behind. Note `:latest` on this repo is the same digest as `main` (rolling dev build), so pin a `vYYYY.M.D` tag. Rollback image `nousresearch/hermes-agent:rollback-2026.7.1`; volume backup in volume `hermes-data-backup-20260820` (`hermes-data-20260820.tgz`, 29MB). **Matrix pairing:** hermes denies unknown senders by default — approve with `docker exec hermes hermes pairing approve matrix <CODE>` (`@jeremy:hobbs.farm` approved 2026-08-20; state persists in the volume). **RESOLVED 2026-08-21 by moving off ChatGPT entirely.** Its Codex refresh token had expired (`Could not validate your refresh token`) and, with **no API key set for any provider**, that was the only auth path — both Matrix and `/api/ask` failed. Verified NOT an upgrade regression (the rollback image failed identically on the same volume). Rather than re-auth interactively, hermes was pointed at the **local llamacpp** endpoint: `model.provider=llamacpp`, `model.default=qwen3.8-64k`, `model.base_url=http://192.168.86.150:8081/v1`. No tokens, no TTY, no subscription. Config backups: `/opt/data/config.yaml.bak-20260820-ollama` and `.bak-20260821-llamacpp`. **`hermes model` is interactive-only — use `hermes config set` for non-interactive changes.** hermes also **hard-refuses any model advertising <64K context** ("Choose a model with at least 64K context"), which is why the served model is a 64k build, not 32k.
- **llamacpp**: :8081 — **the inference backend for hermes as of 2026-08-21.** `ghcr.io/ggml-org/llama.cpp` **pinned by digest** `sha256:995fe9f2...`. Serves `qwen3.8-64k` (alias) from ollama's GGUF blob, mounted read-only. Compose: `compose/gpu-machine/llamacpp/`. Replaced ollama as the server after a same-model/same-context benchmark: **ollama 15.8 tok/s vs llama.cpp 39.7 tok/s (2.4x)**, because `--cache-type-k/v q8_0` halves the KV cache (23.2GB -> 18.6GB) so the model fits the 3090 instead of spilling layers to CPU. Ollama has no KV-quantization knob. The image's own HEALTHCHECK probes :8080 and can never pass here, so it is `disable: true`d — liveness is an external GET of `:8081/health`.
- **friday-voice**: :8320 — push-to-talk console. Calls speaches (`host.docker.internal:8000`) + hermes (`docker exec hermes hermes -Q` via wrapper). Public: **coach.hobbs.farm** → :8320. Compose: `compose/gpu-machine/friday-voice/`.
- Ollama: :11434 (qwen2.5:14b, Hermes-4.3-36B-GGUF, others)
- Ollama: **TWO separate instances on this box, easy to confuse** —
- **Native Windows** `ollama.exe` v0.32.15, listens on `::11434` -> reachable as **`192.168.86.150:11434`**. Holds `qwen3.8`, `qwen3.8-64k`, `gemma4`. This is the LAN-visible one and the source of the GGUF that llamacpp serves.
- **Inside WSL** (`/usr/local/bin/ollama serve`, systemd unit, **enabled+autostart**) v0.15.5, relayed to Windows `127.0.0.1:11434` by `wslrelay.exe`. Holds `qwen2.5:14b`, `hf.co/NousResearch/Hermes-4.3-36B-GGUF:Q4_K_M`, `qwen3.6`, `glm-4.7-flash`, `qwen3-coder:30b`, `gpt-oss:20b`, `llama3.1`. **`host.docker.internal:11434` from a Docker Desktop container resolves to THIS one**, not the native one — that cost real debugging time.
- Both share the one physical 3090. Native ollama env (User scope): `OLLAMA_KEEP_ALIVE=30s`, `OLLAMA_MAX_LOADED_MODELS=1`, `OLLAMA_NUM_PARALLEL=1` — keep-alive was deliberately dropped from `-1` back to `30s` once llamacpp became the server, so ollama stops pinning ~22GB of VRAM. **These are read from the registry only by processes launched fresh (e.g. at login); a process started from an existing shell inherits that shell's env instead.**
- **speaches (voicecoach-speaches-gpu) REMOVED 2026-08-21** to free ~3.3GB VRAM for the llama.cpp benchmark, per user ("they can be rebuilt later"). **Image and the `voicecoach-speaches_speaches-models` volume were kept**, so rebuild is just `docker compose -f 'C:\Users\Jeremy\projects\docker\compose\gpu-machine\voicecoach-speaches\docker-compose.yml' up -d` with no model re-download. **FRIDAY's voice in/out (STT+TTS) is down until then; the text `/api/ask` path works.** Its compose sets `WHISPER__TTL=-1`, which is why it pinned VRAM permanently — consider a finite TTL on rebuild.
### Port Conflict Notes
@@ -234,7 +244,7 @@ After ANY infrastructure change (deploy, remove, reconfigure, add service, chang
2. **MEMORY.md** — Add/update relevant memory entries
3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus
4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
5. **Farm Manager links page** — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)
5. **Farm Manager links page** (last audited 2026-08-20: dropped dead Clippy/Bob/Aaron links, added FRIDAY Voice + LinkedStorm + Rongorongo and a new `GPU Machine (192.168.86.150)` category; `hermes` intentionally absent — it publishes no ports. Served path is `/app.js`, not `/static/app.js`.) — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)
Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.
@@ -302,7 +312,7 @@ All persistent data on NAS at `/mnt/docker-data/`:
9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) removed (replaced by FRIDAY Voice on .150) — **but see correction: the VoiceCoach compose stacks on .192 were only *stopped* in July, not `down`'d, and reappeared (~3GB) on the 2026-08-05 reboot; properly torn down 2026-08-05 (`docker compose down` on `voicecoach` + `voicecoach-v2`), node mem 9.0→6.7GB used.** Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node.
12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) removed (replaced by FRIDAY Voice on .150) — **but see correction: the VoiceCoach compose stacks on .192 were only *stopped* in July, not `down`'d, and reappeared (~3GB) on the 2026-08-05 reboot; properly torn down 2026-08-05 (`docker compose down` on `voicecoach` + `voicecoach-v2`), node mem 9.0→6.7GB used.** — **this is WRONG, see 2026-08-20: only `voicecoach-v2` is gone; the 6-container `voicecoach` project is still running on .192.** Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node.
13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
@@ -310,7 +320,7 @@ All persistent data on NAS at `/mnt/docker-data/`:
17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`).
18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router.
19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.
20. **UniFi OS Server probe port**: `unifi-os-server` on .100 serves HTTPS on host port **11443** (container 443), not 8443. Blackbox probe fixed 2026-07-13. solo.hobbs.farm returns 401 (Traefik basicAuth) when healthy — probed via `http_2xx_or_401` module.
20. **UniFi OS Server probe port** (historical): the `unifi-os-server` container on .100 served HTTPS on host port **11443** (container 443), not 8443. **That container was removed 2026-10-05** (duplicate controller) — the sole UniFi controller is now the Podman instance on **.150** (UI https://192.168.86.150:11443, inform :8080). The live Prometheus config **did** carry a blackbox probe for `https://192.168.86.100:11443` (line 132) — it was **removed 2026-10-06** (backup `prometheus.yml.bak-20261006-unifi-probe-remove`) and Prometheus hot-reloaded, since nothing listens there now. **No probe added for the .150 controller** because that box is a workstation that is sometimes off (it would false-alarm); the switch itself (192.168.86.159) could be ICMP-probed instead if device-down alerting is wanted. (Note: the `UnifiDeviceDown` alert in `alerts.yml` keys on `unifi_device_up`, which needs a `unifi-exporter` that is NOT deployed — the metric doesn't exist, so that rule is inert, not firing.)
21. **OpenVAS pg_upgrade loop saturated bart's HDD (2026-07-15)**: `immauss/openvas:latest` looped on the PG 13→15 auto-upgrade for ~21h (158 restarts) — each cycle re-copied the 21GB DB, pinning sda at ~90% util/40% iowait and making Jellyfin (same disk) hang. The upgrade itself succeeded every time but the final `mv /data/database /data/database_13_old` failed with "Device or resource busy" because the openvas-db volume is mounted **at** `/data/database` (can't rename a mountpoint). Fixed by wiping the DB (`docker volume rm openvas_openvas-db`) and letting it initdb fresh on PG15 — scan history lost, feeds re-synced. **On the next major PG bump this will recur** unless the upgrade is done out-of-band (content-level swap) or the volume is remounted at `/data`. General rule: a service "hanging" on bart usually means another container is eating the single HDD — check `iostat -x`/`pidstat -d` first. **OpenVAS fully decommissioned 2026-07-19** (moot unless redeployed).
22. **Jellyfin SQLite-on-NFS lockups (fixed 2026-07-19)**: jellyfin.db (WAL mode) on `/mnt/docker-data/configs/jellyfin` hit escalating `SQLite Error 5: 'database is locked'` (22→33→125/day), DB commands blocking 60-102s → healthcheck flaps + playback stalls ("Unexpected end of request content" on /Sessions/Playing/Progress). Config moved to LOCAL `/var/lib/jellyfin` (old copy left on NFS as backup). Same class as Known Issue #16.
@@ -322,4 +332,14 @@ All persistent data on NAS at `/mnt/docker-data/`:
27. **Matrix auth is delegated to MAS -- Synapse does NOT serve /login any more (2026-08-20)**: with `matrix_authentication_service.enabled: true`, Synapse answers `M_UNRECOGNIZED` on `/_matrix/client/*/login`. The **reverse proxy must route the three compat paths to MAS or nobody can sign in**: `login`, `logout`, `refresh` (verified: MAS returns 200/405 on those and 404 on `register`/`whoami`, which stay on Synapse). Implemented as the `mess-auth-compat` router in `/mnt/docker-data/traefik/dynamic/temp-routes.yml` with explicit `priority: 1000` so it beats the catch-all `mess` router. **Anything that talks to `192.168.86.192:8008` directly bypasses Traefik and therefore cannot log in with a password** -- token-based bots (spirit, frigate, alertbot) are unaffected because they never call /login, but `@clippyauto` (n8n) and `@pluto` historically re-login with a password (17 and 15 accumulated devices). They currently run on migrated tokens; if either ever needs a fresh password login, point it at `https://mess.hobbs.farm` instead of `:8008`.
28. **MAS image tags are NOT `v`-prefixed** -- `ghcr.io/element-hq/matrix-authentication-service:1.23.0` is correct, `:v1.23.0` does not exist. This is the **opposite** of Synapse (`ghcr.io/element-hq/synapse:v1.159.0`, where `:1.159.0` 404s). The MAS image is also **distroless** -- no shell, so a container healthcheck is impossible; liveness comes from the blackbox probe of `http://192.168.86.192:8021/health`. Note `/health` lives on the internal listener only: `https://auth.hobbs.farm/health` returns 404 by design, so the public probe targets `/.well-known/openid-configuration`.
29. **MAS config holds unrecoverable signing keys**: `/mnt/docker-data/configs/mas/config.yaml` (mode 640) contains the RSA/EC signing keys and the `matrix.secret` shared with Synapse. Regenerating them invalidates every session. It is **not yet backed up to Vaultwarden** because the vault master password is currently unknown (see memory `vault-master-password-stale`) -- resolve that and store it. Also note `http.issuer`/`public_base` (`https://auth.hobbs.farm/`) are baked into session state; changing the hostname later invalidates sessions.
29. **MAS config holds unrecoverable signing keys**: `/mnt/docker-data/configs/mas/config.yaml` (mode 640) contains the RSA/EC signing keys and the `matrix.secret` shared with Synapse. Regenerating them invalidates every session. It is **not yet backed up to Vaultwarden** because the vault master password is currently unknown (see memory `vault-master-password-stale`) -- resolve that and store it. Also note `http.issuer`/`public_base` (`https://auth.hobbs.farm/`) are baked into session state; changing the hostname later invalidates sessions.
30. **OpenClaw agent images were built from `openclaw@latest` and rotted for months (fixed 2026-08-20)**: margaret and spirit sat on **2026.5.27** while upstream was on **2026.7.1-2** — same failure mode as the abandoned `matrixdotorg/synapse:latest` (Known Issue #25). Worse, **the Dockerfiles existed nowhere** — not in the repo, not on any node — and had to be reconstructed from `docker history` layer output. They now live at `compose/hf-pdocker-01/ai-agents/{margaret,spirit}/Dockerfile` (staged on the NAS at `/mnt/docker-data/compose/hf-pdocker-01/ai-agents/`) with `ARG OPENCLAW_VERSION` **pinned**; bump it deliberately. Rebuild: `cd /mnt/docker-data/compose/hf-pdocker-01/ai-agents && docker build -t margaret-openclaw:<ver> -t margaret-openclaw:latest ./margaret` (same for `./spirit` → `spirit-ops`). npm dist-tags to check: `latest`, `extended-stable`, `beta`. **Upgrading migrates the state dir one-way** (JSON sidecars → shared SQLite; originals archived as `*.migrated` under `/mnt/infrastructure/{margaret,spirit}/`), so a rollback to `*:rollback-2026.5.27` also needs those `.migrated` files restored. Config backups: `/mnt/infrastructure/{margaret,spirit}/openclaw.json.bak-20260820-preupgrade`.
31. **`192.168.86.27` is dead and several configs still point at it (2026-08-20)**: spirit's *primary* model was `vllm-local/qwen` at `192.168.86.27:8001`, so **every** message burned ~12s timing out before failing over to Anthropic (2,913 consecutive primary-model failures). margaret had the same host as a `vllm-local/hermes` provider. Both `vllm-local` provider blocks were deleted; both agents are now `anthropic/claude-sonnet-4-6` primary with `google/gemini-2.5-flash` fallback. `voicecoach-openclaw` **still** points at `192.168.86.27:11434` with no fallbacks. **There is no host at .27** — grep any new agent config for it. Related: margaret listed `google/gemini-2.5-flash` as a fallback while defining **no** google provider or auth profile, so it could never have fired; a google provider block (api `google-generative-ai`, key verified against the live models list) was added to match spirit's working shape.
32. **CORRECTED 2026-08-21 — there are TWO ollama instances on .150, which is why the model lists disagreed**: a **native Windows** `ollama.exe` (v0.32.15) on `192.168.86.150:11434` holding only `gemma4` + the new `qwen3.8*`, and one **inside WSL** (v0.15.5, systemd, autostart) relayed to `127.0.0.1:11434`, holding `qwen2.5:14b`, `Hermes-4.3-36B-GGUF`, `qwen3.6`, `qwen3-coder:30b`, `gpt-oss:20b`, `llama3.1`. So the older docs were **right** — those models exist, just in the WSL instance. An earlier note here claiming "only gemma4" was measuring the native one alone. **`host.docker.internal:11434` from a Docker Desktop container hits the WSL instance; the LAN IP hits the native one.** Both share the single 3090, so either can starve the other. spirit's `ollama` provider still advertises a nonexistent `qwen3.5` (harmless, not in its chain). **LinkedStorm on .192 points at `192.168.86.150:11434` for `qwen2.5:14b` — that model is NOT on the native instance, so LinkedStorm's LLM calls are probably 404ing. Still unverified.**
33. **Docker Desktop on .150 died twice while loading a large model (2026-08-21)**: the WSL2 VM had **no `.wslconfig`**, so it defaulted to 50% of host RAM = **15GB** shared by the Ubuntu distro, the docker-desktop VM and podman-uosserver. Loading llama.cpp's 16.8GB GGUF alongside the running containers **OOM-killed hermes and friday-voice (exit 137)**, and on the first attempt took the whole Docker Desktop backend down — it went `unable to start` -> HTTP 500 on every API call -> engine pipe missing entirely. **Recovery that worked:** kill all `Docker Desktop`/`com.docker` processes, `wsl --terminate docker-desktop`, then start Docker Desktop **before** UniFi OS Server (the pipe-conflict rule in issue #13 still applies — UniFi went down in the crash and had to be restarted after). Mitigation applied: `C:\Users\Jeremy\.wslconfig` now sets `memory=22GB`, `swap=8GB` — **this needs `wsl --shutdown` or a reboot to take effect, and `wsl --shutdown` kills the Claude Code session too**, so it has NOT been applied yet. Until it is, expect OOM churn when a big model loads.
34. **Onyx self-registration was open to the internet (fixed 2026-09-21)**: `onyx.hobbs.farm` is publicly reachable and Onyx runs `AUTH_TYPE=basic` with **no** `REQUIRE_EMAIL_VERIFICATION` and **no** `VALID_EMAIL_DOMAINS` — so `POST /api/auth/register` let anyone on the internet create an account. External scanners were already probing the host (a DigitalOcean IP hit `/` on 2026-09-21). No abuse had occurred: only 2 accounts existed (`glyptic@gmail.com` + Onyx's internal `anonymous@onyx.app`). **Fix: a Traefik router `onyx-register-lockdown` (`priority: 1000`, beats the catch-all `onyx` router) matching `PathPrefix(/api/auth/register)` behind an `ipAllowList` middleware limited to `192.168.86.0/24` + `127.0.0.1/32`.** Verified by temporarily narrowing the range: register returned **403** from a non-allowlisted source while `/api/auth/login` (405) and `/` (200) were unaffected; range then restored. Config `/mnt/docker-data/traefik/dynamic/onyx.yml` (backup `.bak-20260920-195851`). **To add a user: register from the LAN, or widen the `sourceRange`.** Note `VALID_EMAIL_DOMAINS` was rejected as the control because it is domain-level only — `gmail.com` would admit any Gmail user. Same class as Known Issue #23 (open Matrix registration).
35. **The live Prometheus config is `/mnt/docker-data/configs/prometheus/prometheus.yml` — the copy at `compose/hf-pdocker-01/monitoring/prometheus.yml` is STALE and misleading (2026-09-21)**: the compose-dir copy still lists decommissioned targets, including Onyx at the wrong node (`192.168.86.167:3005`, bart) and OpenVAS (`:9392`). Neither is in the live config, so neither was ever probed — grepping the compose copy to answer "is X monitored?" gives the wrong answer. Onyx had **no probe at all** until one was added to the live file on 2026-09-21 (`http://192.168.86.192:3005`, now `probe_success=1`). Backup `prometheus.yml.bak-20260920-*`. The live file is also the better changelog: it records the voicecoach (2026-08-24) and FRIDAY Voice / speaches (2026-08-26) teardowns that CLAUDE.md still described as live.
36. **SSH key auth to hf-pdocker-01 is broken — password only (2026-09-21)**: all of `~/.ssh/id_docker_server`, `id_bart`, `id_nas` are rejected by `192.168.86.192` (`Permission denied (publickey,password)`), while `id_docker_server` still works for `.100` and `id_bart` for bart. Connect with `sshpass -p '&Rip8a115' ssh -o PubkeyAuthentication=no glyptic@192.168.86.192`. **Any key-based automation targeting .192 is silently failing** — check `~/.ssh/authorized_keys` on that node (likely lost when the host was rebuilt/restored).