infra: 2026-10-26 version audit, upgrades, UniFi re-removal

- Synapse v1.163.2, MAS 1.27.1, Traefik v3.7, node-exporter v1.12.1 (all nodes), cadvisor -> ghcr.io v0.60.6, alertmanager-receiver 2026.9.23, Gitea 1.26.4, Jellyfin 12.3, Seerr v3.5.1 - Watchtower -> nickfedor/watchtower fork, twice daily (04:00/16:00) - Monitoring legacy node-exporter gated behind a compose profile - unifi-os-server on .100 removed again; compose file renamed so it cannot be revived - CLAUDE.md: Known Issues #40-43, service tree updates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQhgEUqcaV4WM2S4hfXgHD
infra: pin hermes, add llamacpp backend + pinned agent Dockerfiles
2026-10-26 16:27:43 -05:00 · 2026-10-06 19:06:06 -05:00 · 2026-10-06 19:06:05 -05:00 · 2026-08-20 16:36:27 -05:00 · 2026-08-20 15:21:21 -05:00 · 2026-08-20 15:18:10 -05:00
21 changed files with 1288 additions and 109 deletions
@@ -23,14 +23,15 @@ This repository manages a **homelab infrastructure** with self-hosted services r
 hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
 ├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
 ├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
-├── [REMOVED] clip-service -- decommissioned 2026-07-13 (CLIP embeddings; ~2GB RAM freed, no config references from frigate-notify existed)
+├── [COMPOSE] clip-service (:5200) -- **RUNNING, and actively used.** CORRECTED 2026-09-21: the 2026-07-13 decommission note below was wrong for the current deployment — it was redeployed and the *live* `frigate-notify/docker-compose.yml` sets `CLIP_SERVICE_URL=http://localhost:5200`, with live `app/recognition.py` calling `/embed`. ~1.2GB RSS. **Do not remove without first rewiring frigate-notify's recognition pipeline.**
 ├── [COMPOSE] Home Assistant (:8123)
-├── [COMPOSE] ESPHome (:6052)
+├── [COMPOSE] ESPHome (:6052) -- manages **Pool Value Controls** (ESP8266 at static **192.168.86.6**, drives pool pumps/valves; only ESPHome device registered in HA). Also polelightpond (.14) and living_room_lights (.153) configs, NOT in HA. See Known Issue #39.
 ├── [COMPOSE] Mosquitto (:1883, :8883)
-├── [COMPOSE] Synapse/Matrix (:8008) + synapse-db (postgres) + wellknown (:8010)
+├── [COMPOSE] Synapse/Matrix (:8008) + wellknown (:8010) + synapse-db (**postgres:16-alpine, LOCAL /var/lib/synapse-db, C collation**) -- Synapse data on LOCAL /var/lib/synapse. Migrated SQLite->PostgreSQL 2026-08-20 (required by syn2mas). **Auth is delegated to MAS** -- Synapse no longer serves /login.
 ├── [COMPOSE] **MAS / Matrix Authentication Service (:8020 web, :8021 health) + mas-db (postgres:16-alpine, LOCAL /var/lib/mas-db)** -- OAuth2/OIDC provider for Matrix. Public: auth.hobbs.farm. Deployed 2026-08-20 to enable QR sign-in (MSC4108). QR sign-in **verified working end-to-end 2026-08-20**. **Every Matrix login now depends on this service.**
 ├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
-├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
+├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops]) -- **OpenClaw 2026.7.1-2, images PINNED via Dockerfile ARG** (`compose/hf-pdocker-01/ai-agents/{margaret,spirit}/Dockerfile`). Both primary `anthropic/claude-sonnet-4-6`, fallback `google/gemini-2.5-flash`.
-├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
+├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090) -- `traefik:v3.7` (3.7.16, bumped from v3.6 2026-10-26; 3.7 migration notes are k8s-only for us)
 ├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192.
 ├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
 ├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
@@ -38,35 +39,39 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
 ├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
 ├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
 ├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
 ├── [COMPOSE] **Onyx (RAG/search) -- 10 containers, `/mnt/docker-data/compose/hf-pdocker-01/onyx`.** NOT decommissioned (the bart entry below refers to an older, separate deployment). Public: **onyx.hobbs.farm** -> nginx :3005. Custom build: `gitea_patch.py` + `unstructured_patch.py` index Gitea (`git.hobbs.farm/api/v1`) and call unstructured-api on .100:8000. LLM -> `192.168.86.150:18020/v1` (`qwen3.8-27b`) — **dead while .150 is down.** `onyx-opensearch-1` **stopped 2026-09-21** (`ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false`, held no Onyx data, nothing `depends_on` it; freed ~1.1GB). **2026-10-10: it came back on its own after .192's crash/reboot (1.5GB) — now stopped, `restart=no`, and gated behind compose `profiles: ["opensearch"]` so neither reboots nor `compose up` revive it.** Restart with `docker compose --profile opensearch up -d opensearch` if the flag is ever flipped.
 ├── [COMPOSE] hermes -- **moved here from .150** (host network, `/mnt/docker-data/compose/hf-pdocker-01/hermes`, pinned `v2026.8.18`). Points at `http://192.168.86.150:18020/v1` model `qwen3.8-27b`; that endpoint is **dead**, so every call burns 200s x3 retries and fails. Note this is a different host:port than the `.150:8081` llamacpp setup documented in the GPU Machine section.
 ├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
 ├── [COMPOSE] synapse-admin -- Synapse admin UI
-├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
+├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
 ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
 ├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (4x daily 4am/10am/4pm/10pm, log ~/frigate-restart.log)
 ├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)
 bart (15GB RAM, i5-4570T, HDD)
-├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12.
+├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, **Seerr** -- replaced Jellyseerr 2026-10-17, `seerr/seerr:v3.5.1`, config LOCAL /var/lib/seerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin on official STABLE 12.x, pinned to the dated tag `jellyfin/jellyfin:12.3.20261023-013714` (12.3.0, 2026-10-26).** History: ran the 2026060106 unstable build from 2026-07-14 because the DB had been migrated to the v12 schema (10.11.x cannot read it); stable 12.0 shipped 2026-09-08, moved to 12.2 on 2026-10-17, 12.3 on 2026-10-26 (DB backup `/var/lib/jellyfin-backup-20261026-pre123`). **Never go below 12.**
-├── [REMOVED] Onyx -- fully decommissioned 2026-07-13 (had been down since ~2026-06-08 unnoticed; containers, volumes, images removed)
+├── [REMOVED] Onyx -- this *bart* deployment was decommissioned 2026-07-13. **A different, newer Onyx now runs on hf-pdocker-01 (see above) — 'Onyx is gone' is false.**
 ├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15.
 ├── [COMPOSE] Farm Agent (:8889)
-├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
+├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
 ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
-├── [COMPOSE] Watchtower
+├── [COMPOSE] Watchtower -- `nickfedor/watchtower` (maintained fork; containrrr is archived), runs **twice daily 04:00 + 16:00** (`WATCHTOWER_SCHEDULE=0 0 4,16 * * *`)
 hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
-├── [COMPOSE] UniFi OS Server (container `unifi-os-server`, HTTPS on host :11443)
+├── [REMOVED] UniFi OS Server -- the `unifi-os-server` container (ghcr.io/lemker/unifi-os-server, v5.1.15) was decommissioned 2026-10-05. It was a **duplicate/idle** controller -- the real UniFi controller is the **Podman UniFi OS Server on the Windows box (.150)**, which manages all UniFi devices (e.g. `downstairsswitch` US-24-250W at 192.168.86.159, inform `http://192.168.86.150:8080/inform`). Container + image + network removed; ports 11443/8080/3478/etc freed on .100. **Data preserved** at `/var/lib/unifi-os/*` (971M local) + `/mnt/docker-data/configs/unifi-os/persistent` (NAS); compose dir kept for reference. **Only .150 manages UniFi, by design.** **2026-10-26: it was found RUNNING again** (the 10-05 removal never stuck -- journal shows it up again later that same day, then recreated from its compose dir 2026-10-17 21:06Z). Removed again (container, `unifi_default` network, `ghcr.io/lemker/unifi-os-server` + stale `linuxserver/unifi-network-application` images) and the compose file renamed to `docker-compose.yml.decommissioned-20261026` + `DECOMMISSIONED.txt` so a stray `compose up` cannot revive it. Data still preserved.
 ├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
 ├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
 ├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
 ├── [COMPOSE] Vaultwarden, MCPO, Trivy
 ├── [COMPOSE] SearXNG (`searxng/searxng:latest`) -- found running 2026-10-26, previously undocumented here
 ├── [COMPOSE] unstructured-api (:8000, `downloads.unstructured.io/unstructured-io/unstructured-api:latest`) -- used by Onyx's `unstructured_patch.py` on .192
 ├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget)
 ├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
 ├── [COMPOSE] CouchDB (:5984) -- Obsidian LiveSync backend for the Second Brain vault. LAN-only. LOCAL disk (NOT NFS).
 ├── [COMPOSE] Farm Agent (:8889)
-├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
+├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
 ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
-├── [COMPOSE] Watchtower
+├── [COMPOSE] Watchtower -- `nickfedor/watchtower` (maintained fork; containrrr is archived), runs **twice daily 04:00 + 16:00** (`WATCHTOWER_SCHEDULE=0 0 4,16 * * *`)
 ```
@@ -145,7 +150,9 @@ compose/             # Standalone Compose definitions (deployed per-node)
 - Home Assistant: `yoda.hobbs.farm`
 - Vaultwarden: `vault.hobbs.farm`
 - Gitea: `git.hobbs.farm`
- FRIDAY Voice: `coach.hobbs.farm` (replaced VoiceCoach 2026-07-05; runs on .150 :8320)
+- FRIDAY Voice: `coach.hobbs.farm` — **DECOMMISSIONED 2026-08-26; `coach.hobbs.farm` returns 404, which is EXPECTED, not an outage.** CORRECTED 2026-09-21: the "UP as of 2026-08-20" text previously here was overtaken by events. The live Prometheus config records the `voicecoach` teardown (2026-08-24) and the FRIDAY Voice + speaches teardown (2026-08-26) — routing and monitoring sides done, with "container/volume teardown on .150 still pending manual action next Windows boot". The 6-container `voicecoach` project is indeed gone from .192 (verified 2026-09-21). Its `.150` images/volumes may still exist. See memory `friday-voice`.
 - MAS (Matrix auth): `auth.hobbs.farm` -- OAuth2/OIDC issuer. **If this is down, nobody can log into Matrix** (existing tokens keep working).
 - Onyx (RAG/search): `onyx.hobbs.farm` -- **self-registration is LAN-only** (Traefik `onyx-register-lockdown` router). See Known Issue #34.
 - Rongorongo: `rongorongo.hobbs.farm`
 - LinkedStorm: `linkedstorm.hobbs.farm`
@@ -158,6 +165,7 @@ compose/             # Standalone Compose definitions (deployed per-node)
 - ESPHome: :6052
 - Mosquitto: :1883
 - Matrix/Synapse: :8008
 - MAS: :8020 (web/oauth/compat), :8021 (health only -- /health is NOT on :8020)
 - Well-known (Matrix federation): :8010
 - Guacamole: :8199
 - Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
@@ -167,16 +175,18 @@ compose/             # Standalone Compose definitions (deployed per-node)
 - InfluxDB: :8086
 - Blackbox Exporter: :9115
 - matrix-alertmanager-receiver: :9094
- AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0)
+- AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0). OpenClaw **2026.7.1-2** (upgraded 2026-08-20 from 2026.5.27). Matrix: `@margaret:hobbs.farm` / `@spirit:hobbs.farm`, token auth against `http://192.168.86.192:8008`, encryption off, `requireMention: false`. Rollback images kept as `margaret-openclaw:rollback-2026.5.27` / `spirit-ops:rollback-2026.5.27`.
 - Onyx: :3005 (onyx-nginx). Public onyx.hobbs.farm. `onyx-opensearch-1` deliberately stopped 2026-09-21.
 - hermes: no published port (host network) -- see service tree; its LLM endpoint is currently dead.
 - Farm Manager: :8888 (dashboard + API)
 - Farm Agent: :8889
 - Rongorongo: :8181 (rongorongo-web nginx; reverse-proxies /api/* to rongorongo-api Flask backend on the internal `rongorongo` bridge net). State + images at /mnt/docker-data/configs/rongorongo/data/. Public URL gated by Traefik basicAuth (creds in Vaultwarden).
 - LinkedStorm: :8300 (pipeline/VAD+STT+speaker-ID), :8310 (API/dashboard), cron (no port). Public: linkedstorm.hobbs.farm -> :8310. Source: /opt/linkedstorm (rsync from WSL). Data: /mnt/docker-data/configs/linkedstorm/. STT via Speaches GPU machine :8000, LLM via Ollama :11434 (qwen2.5:14b).
- ~~VoiceCoach~~ -- fully decommissioned 2026-07-05, replaced by FRIDAY Voice on .150 (see GPU Machine section + memory `friday-voice`)
+- ~~VoiceCoach~~ -- replaced by FRIDAY Voice on .150 (see GPU Machine section + memory `friday-voice`). **CORRECTION 2026-08-20: the teardown claimed below never actually happened.** All 6 containers of compose project `voicecoach` (`voicecoach-web` :8085, `-pipecat`, `-speaches`, `-speechbrain`, `-openclaw` :3100, `-livekit` :7880-7881 + 50000-50100/udp) are **running on .192, up since ~2026-08-06**. Earlier text said both projects were `docker compose down`'d on 2026-08-05; that is false for `voicecoach` (`voicecoach-v2` is genuinely gone). ~1GB RAM. `voicecoach-openclaw` was upgraded to OpenClaw 2026.7.1-2 on 2026-08-20 via `docker compose up -d --no-deps voicecoach-openclaw` (single service only — does not resurrect anything, the rest were already up). Its model is `ollama-local/qwen2.5:14b` at the **dead** `192.168.86.27:11434` with **no fallbacks**, so it cannot serve a turn. **Decide: fix its model chain or actually tear the stack down.**
 **hf-pdocker-02 (192.168.86.100):**
 - Farm Agent: :8889
- UniFi OS Server: :11443 (HTTPS, container 443)
+- ~~UniFi OS Server: :11443~~ -- **REMOVED 2026-10-05** (duplicate controller; see service tree). The UniFi controller lives on **.150 (Podman), UI at https://192.168.86.150:11443, inform :8080**.
 - Gitea: :3003 (web), :2222 (SSH)
 - Semaphore: :3004
 - Wiki.js: :3002
@@ -195,13 +205,18 @@ compose/             # Standalone Compose definitions (deployed per-node)
 - Jellyseerr: :5055
 **GPU Machine (192.168.86.150, RTX 3090):**
- **NOTE (2026-07-05):** .150 is actually the user's **Windows workstation** (runs the WSL/Claude Code session), not a headless Linux server. It runs **Docker Desktop** (WSL2 backend, RTX 3090 GPU passthrough working) alongside **UniFi OS Server** (Podman). Docker Desktop panics if UniFi/Podman claims `\\.\pipe\docker_engine` first — start Docker Desktop before UniFi OS Server. See memory `docker-desktop-unifi-podman-conflict`.
+- **NOTE (2026-07-05):** .150 is actually the user's **Windows workstation** (runs the WSL/Claude Code session), not a headless Linux server. It runs **Docker Desktop** (WSL2 backend, RTX 3090 GPU passthrough working) alongside **UniFi OS Server** (Podman) — **this is the ONE real UniFi controller** (the duplicate container on .100 was removed 2026-10-05). UI https://192.168.86.150:11443, inform http://192.168.86.150:8080. Manages all UniFi devices incl. `downstairsswitch` (US-24-250W, 192.168.86.159; SSH `admin`/node-password). Docker Desktop panics if UniFi/Podman claims `\\.\pipe\docker_engine` first — start Docker Desktop before UniFi OS Server. See memory `docker-desktop-unifi-podman-conflict`.
 - **FRIDAY Voice replaced VoiceCoach (2026-07-05).** VoiceCoach fully decommissioned (v1+v2 removed from hf-pdocker-01; voice.hobbs.farm route removed). See memory `friday-voice`.
 - [COMPOSE] **FRIDAY Voice stack (all on .150 Docker Desktop, via `docker.exe -c desktop-linux`):**
-  - **voicecoach-speaches-gpu**: :8000 — GPU Whisper large-v3-turbo (STT) + Kokoro-82M af_nova (TTS), CUDA float16. Compose: `compose/gpu-machine/voicecoach-speaches/`.
+  - **voicecoach-speaches-gpu**: :8000 — GPU Whisper large-v3-turbo (STT) + Kokoro-82M af_nova (TTS), CUDA float16. Compose: `compose/gpu-machine/voicecoach-speaches/`. **REMOVED 2026-08-21 (see the speaches note at the end of this section) — rebuild when voice I/O is wanted again.**
-  - **hermes**: `nousresearch/hermes-agent:latest` gateway — FRIDAY's reasoning agent. Settings transferred from WSL `~/.hermes` into named volume `hermes-data`. Compose: `compose/gpu-machine/hermes/`.
+  - **hermes**: **PINNED `nousresearch/hermes-agent:v2026.8.18`** (upgraded 2026-08-20 from v2026.7.1 / v0.18.0 → v0.20.4). FRIDAY's reasoning agent. Settings in named volume `hermes-data`. Compose: `compose/gpu-machine/hermes/`. Was on `:latest` **with Watchtower disabled**, so nothing ever re-pulled it — it sat 7 weeks / 9 releases behind. Note `:latest` on this repo is the same digest as `main` (rolling dev build), so pin a `vYYYY.M.D` tag. Rollback image `nousresearch/hermes-agent:rollback-2026.7.1`; volume backup in volume `hermes-data-backup-20260820` (`hermes-data-20260820.tgz`, 29MB). **Matrix pairing:** hermes denies unknown senders by default — approve with `docker exec hermes hermes pairing approve matrix <CODE>` (`@jeremy:hobbs.farm` approved 2026-08-20; state persists in the volume). **RESOLVED 2026-08-21 by moving off ChatGPT entirely.** Its Codex refresh token had expired (`Could not validate your refresh token`) and, with **no API key set for any provider**, that was the only auth path — both Matrix and `/api/ask` failed. Verified NOT an upgrade regression (the rollback image failed identically on the same volume). Rather than re-auth interactively, hermes was pointed at the **local llamacpp** endpoint: `model.provider=llamacpp`, `model.default=qwen3.8-64k`, `model.base_url=http://192.168.86.150:8081/v1`. No tokens, no TTY, no subscription. Config backups: `/opt/data/config.yaml.bak-20260820-ollama` and `.bak-20260821-llamacpp`. **`hermes model` is interactive-only — use `hermes config set` for non-interactive changes.** hermes also **hard-refuses any model advertising <64K context** ("Choose a model with at least 64K context"), which is why the served model is a 64k build, not 32k.
  - **llamacpp**: :8081 — **the inference backend for hermes as of 2026-08-21.** `ghcr.io/ggml-org/llama.cpp` **pinned by digest** `sha256:995fe9f2...`. Serves `qwen3.8-64k` (alias) from ollama's GGUF blob, mounted read-only. Compose: `compose/gpu-machine/llamacpp/`. Replaced ollama as the server after a same-model/same-context benchmark: **ollama 15.8 tok/s vs llama.cpp 39.7 tok/s (2.4x)**, because `--cache-type-k/v q8_0` halves the KV cache (23.2GB -> 18.6GB) so the model fits the 3090 instead of spilling layers to CPU. Ollama has no KV-quantization knob. The image's own HEALTHCHECK probes :8080 and can never pass here, so it is `disable: true`d — liveness is an external GET of `:8081/health`.
  - **friday-voice**: :8320 — push-to-talk console. Calls speaches (`host.docker.internal:8000`) + hermes (`docker exec hermes hermes -Q` via wrapper). Public: **coach.hobbs.farm** → :8320. Compose: `compose/gpu-machine/friday-voice/`.
- Ollama: :11434 (qwen2.5:14b, Hermes-4.3-36B-GGUF, others)
+- Ollama: **TWO separate instances on this box, easy to confuse** —
  - **Native Windows** `ollama.exe` v0.32.15, listens on `::11434` -> reachable as **`192.168.86.150:11434`**. Holds `qwen3.8`, `qwen3.8-64k`, `gemma4`. This is the LAN-visible one and the source of the GGUF that llamacpp serves.
  - **Inside WSL** (`/usr/local/bin/ollama serve`, systemd unit, **enabled+autostart**) v0.15.5, relayed to Windows `127.0.0.1:11434` by `wslrelay.exe`. Holds `qwen2.5:14b`, `hf.co/NousResearch/Hermes-4.3-36B-GGUF:Q4_K_M`, `qwen3.6`, `glm-4.7-flash`, `qwen3-coder:30b`, `gpt-oss:20b`, `llama3.1`. **`host.docker.internal:11434` from a Docker Desktop container resolves to THIS one**, not the native one — that cost real debugging time.
  - Both share the one physical 3090. Native ollama env (User scope): `OLLAMA_KEEP_ALIVE=30s`, `OLLAMA_MAX_LOADED_MODELS=1`, `OLLAMA_NUM_PARALLEL=1` — keep-alive was deliberately dropped from `-1` back to `30s` once llamacpp became the server, so ollama stops pinning ~22GB of VRAM. **These are read from the registry only by processes launched fresh (e.g. at login); a process started from an existing shell inherits that shell's env instead.**
 - **speaches (voicecoach-speaches-gpu) REMOVED 2026-08-21** to free ~3.3GB VRAM for the llama.cpp benchmark, per user ("they can be rebuilt later"). **Image and the `voicecoach-speaches_speaches-models` volume were kept**, so rebuild is just `docker compose -f 'C:\Users\Jeremy\projects\docker\compose\gpu-machine\voicecoach-speaches\docker-compose.yml' up -d` with no model re-download. **FRIDAY's voice in/out (STT+TTS) is down until then; the text `/api/ask` path works.** Its compose sets `WHISPER__TTL=-1`, which is why it pinned VRAM permanently — consider a finite TTL on rebuild.
 ### Port Conflict Notes
@@ -231,7 +246,7 @@ After ANY infrastructure change (deploy, remove, reconfigure, add service, chang
 2. **MEMORY.md** — Add/update relevant memory entries
 3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus
 4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
-5. **Farm Manager links page** — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)
+5. **Farm Manager links page** (last audited **2026-10-26**: every link probed; removed both FRIDAY Voice links, added Onyx (public + :3005), Guacamole :8199, Synapse Admin :8080, SearXNG :8888, n8n :5678, CouchDB Fauxton; MCPO -> `/docs`, Trivy -> `/healthz` (API only, no UI); Jellyseerr -> Seerr. Groups live in `/mnt/docker-data/configs/farm-manager/groups.json` -- **update them too when containers are added/renamed**. Previous audit 2026-08-20: dropped dead Clippy/Bob/Aaron links, added FRIDAY Voice + LinkedStorm + Rongorongo and a new `GPU Machine (192.168.86.150)` category; `hermes` intentionally absent — it publishes no ports. Served path is `/app.js`, not `/static/app.js`.) — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)
 Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.
@@ -292,21 +307,49 @@ All persistent data on NAS at `/mnt/docker-data/`:
 2. **Telegraf Docker Input**: Disabled due to Docker socket permission issues across nodes
 3. **Watchtower API Version**: Requires `DOCKER_API_VERSION=1.44`
 4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running
-5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
+5. **Frigate cameras**: 192.168.44.0/24 is the **"Camera" VLAN 200** on the Firewalla (gateway 192.168.86.1 / 44.1). Cameras with 44.x IPs MUST sit on Camera-native switch ports. **2026-10-08:** house_north/.51, house_east/.58, house_south/.53, house_west/.52 are wired to downstairsswitch (.159) ports 3/4/18/19 (verified from the switch MAC table 2026-10-11), which are **VLAN 1**, so they can't be reached by routing. They only half-worked because .192's Wi-Fi NIC held a wrong-VLAN 44.x lease. The fix is to make those ports Camera-native (see memory `unifi-estate`). The nodes' `wlp2s0` Wi-Fi NICs must hold **86.x** leases; a 44.x lease there is stale and breaks routing to the real Camera VLAN. **2026-10-10: with the Wi-Fi workaround gone (86.x lease since 10-08), those 4 cameras are now fully DOWN (0 fps, no ping) until the ports are re-profiled.**
 6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
 7. **systemd-resolved DNSStubListener**: disabled on hf-pdocker-01 and hf-pdocker-02 via `/etc/systemd/resolved.conf.d/nostub.conf` (`DNSStubListener=no`) — originally for AdGuard Home's port 53. AdGuard was decommissioned 2026-07-13; the stub-listener change is harmless and left in place (re-enable only if something needs 127.0.0.53).
 8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`.
 9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
 10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
 11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
-12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node.
+12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) removed (replaced by FRIDAY Voice on .150) — **but see correction: the VoiceCoach compose stacks on .192 were only *stopped* in July, not `down`'d, and reappeared (~3GB) on the 2026-08-05 reboot; properly torn down 2026-08-05 (`docker compose down` on `voicecoach` + `voicecoach-v2`), node mem 9.0→6.7GB used.** — **this is WRONG, see 2026-08-20: only `voicecoach-v2` is gone; the 6-container `voicecoach` project is still running on .192.** Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node.
 13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
 14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
 15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
-16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart; jellyfin config → `/var/lib/jellyfin` on bart 2026-07-19 after 60-100s "database is locked" stalls on jellyfin.db caused playback lockups). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
+16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart; jellyfin config → `/var/lib/jellyfin` on bart 2026-07-19 after 60-100s "database is locked" stalls on jellyfin.db caused playback lockups; **synapse data → `/var/lib/synapse` on .192 2026-08-20**, pre-emptively — its 279MB SQLite homeserver.db was on NFS). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
 17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`).
 18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router.
 19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.
-20. **UniFi OS Server probe port**: `unifi-os-server` on .100 serves HTTPS on host port **11443** (container 443), not 8443. Blackbox probe fixed 2026-07-13. solo.hobbs.farm returns 401 (Traefik basicAuth) when healthy — probed via `http_2xx_or_401` module.
+20. **UniFi OS Server probe port** (historical): the `unifi-os-server` container on .100 served HTTPS on host port **11443** (container 443), not 8443. **That container was removed 2026-10-05** (duplicate controller) — the sole UniFi controller is now the Podman instance on **.150** (UI https://192.168.86.150:11443, inform :8080). The live Prometheus config **did** carry a blackbox probe for `https://192.168.86.100:11443` (line 132) — it was **removed 2026-10-06** (backup `prometheus.yml.bak-20261006-unifi-probe-remove`) and Prometheus hot-reloaded, since nothing listens there now. **No probe added for the .150 controller** because that box is a workstation that is sometimes off (it would false-alarm); the switch itself (192.168.86.159) could be ICMP-probed instead if device-down alerting is wanted. (Note: the `UnifiDeviceDown` alert in `alerts.yml` keys on `unifi_device_up`, which needs a `unifi-exporter` that is NOT deployed — the metric doesn't exist, so that rule is inert, not firing.)
 21. **OpenVAS pg_upgrade loop saturated bart's HDD (2026-07-15)**: `immauss/openvas:latest` looped on the PG 13→15 auto-upgrade for ~21h (158 restarts) — each cycle re-copied the 21GB DB, pinning sda at ~90% util/40% iowait and making Jellyfin (same disk) hang. The upgrade itself succeeded every time but the final `mv /data/database /data/database_13_old` failed with "Device or resource busy" because the openvas-db volume is mounted **at** `/data/database` (can't rename a mountpoint). Fixed by wiping the DB (`docker volume rm openvas_openvas-db`) and letting it initdb fresh on PG15 — scan history lost, feeds re-synced. **On the next major PG bump this will recur** unless the upgrade is done out-of-band (content-level swap) or the volume is remounted at `/data`. General rule: a service "hanging" on bart usually means another container is eating the single HDD — check `iostat -x`/`pidstat -d` first. **OpenVAS fully decommissioned 2026-07-19** (moot unless redeployed).
 22. **Jellyfin SQLite-on-NFS lockups (fixed 2026-07-19)**: jellyfin.db (WAL mode) on `/mnt/docker-data/configs/jellyfin` hit escalating `SQLite Error 5: 'database is locked'` (22→33→125/day), DB commands blocking 60-102s → healthcheck flaps + playback stalls ("Unexpected end of request content" on /Sessions/Playing/Progress). Config moved to LOCAL `/var/lib/jellyfin` (old copy left on NFS as backup). Same class as Known Issue #16.
 23. **Matrix registration was wide open to the internet (fixed 2026-08-20)**: `mess.hobbs.farm` is publicly reachable (Traefik, port 443, federation + client API on :8008) and `homeserver.yaml` had `enable_registration: true` + `enable_registration_without_verification: true` — the registration flow was a bare `m.login.dummy`, i.e. **anyone on the internet could create an account on hobbs.farm with no email, captcha, or token**. No abuse had occurred (all 18 accounts are known/expected; newest legit account was @fry 2026-07-05). Fixed by setting both to `false` and `restrict_public_rooms_to_local_users: true` (was `false`, which published the public room directory to any remote server). Backup: `/var/lib/synapse/homeserver.yaml.bak-20260820-registration`. New accounts are still creatable on the box via `docker exec synapse register_new_matrix_user -k <registration_shared_secret>`.
 24. **Synapse database history (SQLite -> local disk -> PostgreSQL, all 2026-08-20)**: Synapse ran on **SQLite on NFS** while CLAUDE.md claimed it used a `synapse-db` postgres that in fact had **0 tables**. First the data was moved to LOCAL `/var/lib/synapse` and the empty postgres deleted; then, because `syn2mas` **only reads PostgreSQL** ("The migration tool only supports reading from PostgreSQL for the Synapse database as well"), a real `synapse-db` was created and the DB ported with `synapse_port_db` (20 users / 93,888 events / 65 devices, verified identical; 2m35s downtime). The PG database MUST have `LC_COLLATE=C` -- verify with `select datcollate from pg_database where datname='synapse'` (PG16 removed the `SHOW lc_collate` GUC). The old SQLite file is still at `/var/lib/synapse/homeserver.db`, now inert; NFS copy at `/mnt/docker-data/configs/matrix/data` is an older backup -- **never remount either**.
 25. **Synapse upgraded 1.148.0 -> 1.159.0 and moved to the element-hq registry (2026-08-20)**: it had been on `matrixdotorg/synapse:latest`, an **abandoned Docker Hub repo** whose `:latest` froze at 1.148.0 (image built 2026-02-24) — a moving tag that silently stopped moving on a publicly-exposed homeserver. Now pinned to `ghcr.io/element-hq/synapse:v1.159.0` (note the **`v` prefix** — `:1.159.0` without it 404s). Upgrade was clean: schema 93 -> 94, all background updates finished in ~5s, 0 errors, 19 users / 93,884 events intact, memory 112MB/1G. None of the v1.149-v1.159 breaking changes applied to us (MSC3861 removal — unused; `quarantined_media_changes` stream writer — workers only; the rest are Debian/Ubuntu packaging). Pre-upgrade backup: `/var/lib/synapse.bak-20260820-pre1159` (646MB) — delete once you're satisfied. **Pinned on purpose: bump it manually and deliberately, since the last moving tag rotted for 6 months unnoticed.** **2026-10-26: bumped v1.159.0 -> v1.163.2** (schema 94 deltas applied in <1s; config contains none of the options affected by the 1.160-1.163 upgrade notes). pg_dump backups taken first: `/var/lib/upgrade-backups-20261026/{synapse-db,mas-db}.dump` on .192. MAS bumped 1.23.0 -> **1.27.1** at the same time.
 26. **`SYNAPSE_SERVER_NAME` in the matrix compose disagrees with homeserver.yaml**: compose passes `SYNAPSE_SERVER_NAME=mess.hobbs.farm` but the real `server_name` is **`hobbs.farm`** (user IDs are `@user:hobbs.farm`). The env var is only read when the image *generates* a config, so it's inert today — but if `homeserver.yaml` ever went missing, the container would generate a fresh config with the wrong `server_name` AND a new signing key, which would break federation and every existing user ID. Left as-is (changing it has no effect while the config exists); just never let the container auto-generate..
 27. **Matrix auth is delegated to MAS -- Synapse does NOT serve /login any more (2026-08-20)**: with `matrix_authentication_service.enabled: true`, Synapse answers `M_UNRECOGNIZED` on `/_matrix/client/*/login`. The **reverse proxy must route the three compat paths to MAS or nobody can sign in**: `login`, `logout`, `refresh` (verified: MAS returns 200/405 on those and 404 on `register`/`whoami`, which stay on Synapse). Implemented as the `mess-auth-compat` router in `/mnt/docker-data/traefik/dynamic/temp-routes.yml` with explicit `priority: 1000` so it beats the catch-all `mess` router. **Anything that talks to `192.168.86.192:8008` directly bypasses Traefik and therefore cannot log in with a password** -- token-based bots (spirit, frigate, alertbot) are unaffected because they never call /login, but `@clippyauto` (n8n) and `@pluto` historically re-login with a password (17 and 15 accumulated devices). They currently run on migrated tokens; if either ever needs a fresh password login, point it at `https://mess.hobbs.farm` instead of `:8008`.
 28. **MAS image tags are NOT `v`-prefixed** -- `ghcr.io/element-hq/matrix-authentication-service:1.23.0` is correct, `:v1.23.0` does not exist. This is the **opposite** of Synapse (`ghcr.io/element-hq/synapse:v1.159.0`, where `:1.159.0` 404s). The MAS image is also **distroless** -- no shell, so a container healthcheck is impossible; liveness comes from the blackbox probe of `http://192.168.86.192:8021/health`. Note `/health` lives on the internal listener only: `https://auth.hobbs.farm/health` returns 404 by design, so the public probe targets `/.well-known/openid-configuration`.
 29. **MAS config holds unrecoverable signing keys**: `/mnt/docker-data/configs/mas/config.yaml` (mode 640) contains the RSA/EC signing keys and the `matrix.secret` shared with Synapse. Regenerating them invalidates every session. It is **not yet backed up to Vaultwarden** because the vault master password is currently unknown (see memory `vault-master-password-stale`) -- resolve that and store it. Also note `http.issuer`/`public_base` (`https://auth.hobbs.farm/`) are baked into session state; changing the hostname later invalidates sessions.
 30. **OpenClaw agent images were built from `openclaw@latest` and rotted for months (fixed 2026-08-20)**: margaret and spirit sat on **2026.5.27** while upstream was on **2026.7.1-2** — same failure mode as the abandoned `matrixdotorg/synapse:latest` (Known Issue #25). Worse, **the Dockerfiles existed nowhere** — not in the repo, not on any node — and had to be reconstructed from `docker history` layer output. They now live at `compose/hf-pdocker-01/ai-agents/{margaret,spirit}/Dockerfile` (staged on the NAS at `/mnt/docker-data/compose/hf-pdocker-01/ai-agents/`) with `ARG OPENCLAW_VERSION` **pinned**; bump it deliberately. Rebuild: `cd /mnt/docker-data/compose/hf-pdocker-01/ai-agents && docker build -t margaret-openclaw:<ver> -t margaret-openclaw:latest ./margaret` (same for `./spirit` → `spirit-ops`). npm dist-tags to check: `latest`, `extended-stable`, `beta`. **Upgrading migrates the state dir one-way** (JSON sidecars → shared SQLite; originals archived as `*.migrated` under `/mnt/infrastructure/{margaret,spirit}/`), so a rollback to `*:rollback-2026.5.27` also needs those `.migrated` files restored. Config backups: `/mnt/infrastructure/{margaret,spirit}/openclaw.json.bak-20260820-preupgrade`.
 31. **`192.168.86.27` is dead and several configs still point at it (2026-08-20)**: spirit's *primary* model was `vllm-local/qwen` at `192.168.86.27:8001`, so **every** message burned ~12s timing out before failing over to Anthropic (2,913 consecutive primary-model failures). margaret had the same host as a `vllm-local/hermes` provider. Both `vllm-local` provider blocks were deleted; both agents are now `anthropic/claude-sonnet-4-6` primary with `google/gemini-2.5-flash` fallback. `voicecoach-openclaw` **still** points at `192.168.86.27:11434` with no fallbacks. **There is no host at .27** — grep any new agent config for it. Related: margaret listed `google/gemini-2.5-flash` as a fallback while defining **no** google provider or auth profile, so it could never have fired; a google provider block (api `google-generative-ai`, key verified against the live models list) was added to match spirit's working shape.
 32. **CORRECTED 2026-08-21 — there are TWO ollama instances on .150, which is why the model lists disagreed**: a **native Windows** `ollama.exe` (v0.32.15) on `192.168.86.150:11434` holding only `gemma4` + the new `qwen3.8*`, and one **inside WSL** (v0.15.5, systemd, autostart) relayed to `127.0.0.1:11434`, holding `qwen2.5:14b`, `Hermes-4.3-36B-GGUF`, `qwen3.6`, `qwen3-coder:30b`, `gpt-oss:20b`, `llama3.1`. So the older docs were **right** — those models exist, just in the WSL instance. An earlier note here claiming "only gemma4" was measuring the native one alone. **`host.docker.internal:11434` from a Docker Desktop container hits the WSL instance; the LAN IP hits the native one.** Both share the single 3090, so either can starve the other. spirit's `ollama` provider still advertises a nonexistent `qwen3.5` (harmless, not in its chain). **LinkedStorm on .192 points at `192.168.86.150:11434` for `qwen2.5:14b` — that model is NOT on the native instance, so LinkedStorm's LLM calls are probably 404ing. Still unverified.**
 33. **Docker Desktop on .150 died twice while loading a large model (2026-08-21)**: the WSL2 VM had **no `.wslconfig`**, so it defaulted to 50% of host RAM = **15GB** shared by the Ubuntu distro, the docker-desktop VM and podman-uosserver. Loading llama.cpp's 16.8GB GGUF alongside the running containers **OOM-killed hermes and friday-voice (exit 137)**, and on the first attempt took the whole Docker Desktop backend down — it went `unable to start` -> HTTP 500 on every API call -> engine pipe missing entirely. **Recovery that worked:** kill all `Docker Desktop`/`com.docker` processes, `wsl --terminate docker-desktop`, then start Docker Desktop **before** UniFi OS Server (the pipe-conflict rule in issue #13 still applies — UniFi went down in the crash and had to be restarted after). Mitigation applied: `C:\Users\Jeremy\.wslconfig` now sets `memory=22GB`, `swap=8GB` — **this needs `wsl --shutdown` or a reboot to take effect, and `wsl --shutdown` kills the Claude Code session too**, so it has NOT been applied yet. Until it is, expect OOM churn when a big model loads.
 34. **Onyx self-registration was open to the internet (fixed 2026-09-21)**: `onyx.hobbs.farm` is publicly reachable and Onyx runs `AUTH_TYPE=basic` with **no** `REQUIRE_EMAIL_VERIFICATION` and **no** `VALID_EMAIL_DOMAINS` — so `POST /api/auth/register` let anyone on the internet create an account. External scanners were already probing the host (a DigitalOcean IP hit `/` on 2026-09-21). No abuse had occurred: only 2 accounts existed (`glyptic@gmail.com` + Onyx's internal `anonymous@onyx.app`). **Fix: a Traefik router `onyx-register-lockdown` (`priority: 1000`, beats the catch-all `onyx` router) matching `PathPrefix(/api/auth/register)` behind an `ipAllowList` middleware limited to `192.168.86.0/24` + `127.0.0.1/32`.** Verified by temporarily narrowing the range: register returned **403** from a non-allowlisted source while `/api/auth/login` (405) and `/` (200) were unaffected; range then restored. Config `/mnt/docker-data/traefik/dynamic/onyx.yml` (backup `.bak-20260920-195851`). **To add a user: register from the LAN, or widen the `sourceRange`.** Note `VALID_EMAIL_DOMAINS` was rejected as the control because it is domain-level only — `gmail.com` would admit any Gmail user. Same class as Known Issue #23 (open Matrix registration).
 35. **The live Prometheus config is `/mnt/docker-data/configs/prometheus/prometheus.yml` — the copy at `compose/hf-pdocker-01/monitoring/prometheus.yml` is STALE and misleading (2026-09-21)**: the compose-dir copy still lists decommissioned targets, including Onyx at the wrong node (`192.168.86.167:3005`, bart) and OpenVAS (`:9392`). Neither is in the live config, so neither was ever probed — grepping the compose copy to answer "is X monitored?" gives the wrong answer. Onyx had **no probe at all** until one was added to the live file on 2026-09-21 (`http://192.168.86.192:3005`, now `probe_success=1`). Backup `prometheus.yml.bak-20260920-*`. The live file is also the better changelog: it records the voicecoach (2026-08-24) and FRIDAY Voice / speaches (2026-08-26) teardowns that CLAUDE.md still described as live.
 36. **SSH key auth to hf-pdocker-01 is broken — password only (2026-09-21)**: all of `~/.ssh/id_docker_server`, `id_bart`, `id_nas` are rejected by `192.168.86.192` (`Permission denied (publickey,password)`), while `id_docker_server` still works for `.100` and `id_bart` for bart. Connect with `sshpass -p '&Rip8a115' ssh -o PubkeyAuthentication=no glyptic@192.168.86.192`. **Any key-based automation targeting .192 is silently failing** — check `~/.ssh/authorized_keys` on that node (likely lost when the host was rebuilt/restored).
 37. **hf-pdocker-01 hard-hung for ~27h (2026-10-09 13:55 → 2026-10-10 17:08 CDT)**: journal simply stops at 13:55 with no OOM, kernel error, or shutdown record; `sar` at 13:50 showed a healthy box (62% mem, 23% swap, load ~4), and .100/bart stayed up (9d uptime), so it was neither memory exhaustion nor a house-wide power event. Came back on kernel 7.0.0-38 (was -34) after a power cycle. Cause unknown — if it recurs, suspect hardware (this box was repaired 2026-03) and consider enabling persistent `kdump`/`netconsole`. Side effects found on 2026-10-10: `onyx-opensearch-1` revived itself (fixed, see Onyx entry). Also fixed then: `ServiceHighMemory` used `container_memory_usage_bytes`, which counts reclaimable page cache, so clip-service false-alarmed at 94% while really at 64% — rule now uses `container_memory_working_set_bytes` (backup `alerts.yml.bak-20261010-workingset`).
 38. **bart shows ~20 nameless `Dead` containers in `docker ps -a`** — orphaned `/var/lib/docker/containers/<id>` dirs for containers whose images Watchtower already deleted. `docker rm` says "No such container". Harmless clutter; clearing them requires stopping dockerd (moving the dirs aside) which restarts the whole media stack, so it was left for a maintenance window.
 39. **[RESOLVED 2026-10-11] Static-IP ESPHome boards were invisible behind UniFi Proxy ARP (found 2026-10-10)**: the `ASUS_F0_2G` WLAN has **Proxy ARP on** (`proxy_arp=1` in every AP's `/etc/hostapd/wifi0ap0.cfg`; it also has `ap_isolate=1` and `mcast.enhance=1`). With Proxy ARP, the AP answers ARP for wireless clients **from a table it learns by snooping DHCP** and stops passing ARP broadcasts through to them. Boards with ESPHome `manual_ip` never DHCP, so after any reconnect nobody answers ARP for them: they associate fine and even announce themselves over mDNS, but are unreachable. **Pool Value Controls (.6, MAC `a4:cf:12:f1:6e:c8`) went dark 2026-10-09 11:04** at a Wi-Fi reconnect on AP .187 and then looped every 15 min on ESPHome's API `reboot_timeout`. Diagnosed by capture on .187: ARP who-has .6 reached the radio and got no reply, while a static neighbor entry made ping and API work at once. **Workaround (live):** systemd unit `static-arp-esphome.service` on .192 pins `.6 -> a4:cf:12:f1:6e:c8 nud permanent` on `enp0s31f6` (HA and ESPHome are bridge-networked on .192, so they use the host's neighbor table). **Real fix (pending, needs controller access on .150):** turn Proxy ARP off on the `ASUS_F0_2G` WLAN, OR switch the boards to DHCP + Firewalla reservations. Once Proxy ARP is off, `systemctl disable --now static-arp-esphome` on .192. A `power_save_mode: none` change was also OTA'd to the board (backup `pool-value-controls.yaml.bak-20261010-powersave`). It did NOT fix this, but it's harmless and was left in. **Any other static-IP Wi-Fi device on that SSID has the same problem.** **RESOLVED 2026-10-11:** Proxy ARP (and `ap_isolate`) were turned off on `ASUS_F0_2G` in the controller (pushed to all APs 11:09–11:18; the `proxy_arp`/`ap_isolate` lines are gone from every AP's hostapd config). `static-arp-esphome.service` was then disabled and deleted on .192. Verified afterwards: .6 resolves by normal ARP (neighbor `REACHABLE`, not `PERMANENT`), ping OK, HA API session established. If a static-IP Wi-Fi device goes dark again, check that Proxy ARP hasn't been switched back on. polelightpond (.14) and living_room_lights (.153) were still unreachable after the fix, so they have some other problem (unpowered or offline).
 40. **Container version audit + upgrade round (2026-10-26)**. Upgraded: Synapse v1.163.2, MAS 1.27.1, Traefik v3.7 (3.7.16), node-exporter v1.12.1 (all 3 nodes), cadvisor v0.60.6, matrix-alertmanager-receiver 2026.9.23, Gitea 1.26.4 (patch only; pg_dump at `~/gitea-db-20261026-pre1264.dump` on .100), Jellyfin 12.3, Seerr v3.5.1, plus re-pulls of every moving tag on .192 (HA 2026.10.4, ESPHome, Mosquitto, Prometheus, Blackbox, Telegraf, nginx, litellm, and the postgres/mysql/redis patch images). Every compose file touched has a `.bak-20261026` next to it on the NAS. **Deliberately NOT done, each needs its own decision:** Onyx `:latest` now = **v5.0.0 (major, released 2026-10-24)**, still on v4.9.2 (no Watchtower on .192, so it only moves on a manual pull); Gitea 1.27.x / **28.x** (major jumps); OpenClaw 2026.7.1-2 -> 2026.10.7 (one-way state migration, see #30); hermes v2026.8.18 -> v0.22.5 (upstream switched to semver tags, and its LLM endpoint is dead anyway); **MySQL 8.0 (guacamole-db, semaphore-db) went EOL April 2026 -> needs an 8.4 LTS migration with a dump/restore**; InfluxDB 1.8 (EOL); Onyx's `postgres:15.2-alpine` (2023 patch level, managed by Onyx's compose); `awesometechnologies/synapse-admin` (stale, maintained fork is `etkecc/synapse-admin`). How to audit: compare each container's `RepoDigests` against registry HEAD digests -- for Docker Hub use the hub.docker.com web API (`/v2/namespaces/<ns>/repositories/<repo>/tags/<tag>` -> `digest`), which does not count against pull limits. `docker buildx imagetools inspect --format` is ignored by the nodes' buildx v0.31.1.
 41. **Docker Hub anonymous pull limit is shared by the whole house** (public IP 98.160.103.126, 100 pulls/hour): on 2026-10-26 every node AND WSL got 429 `toomanyrequests`, meaning Watchtower updates of Docker Hub images were silently failing. Mitigation applied: both Watchtowers moved from a 24h poll interval to a **twice-daily cron schedule** (04:00/16:00), and the archived `containrrr/watchtower` swapped for the maintained drop-in fork `nickfedor/watchtower` (**image name is `nickfedor/`, not `nicholas-fedor/`** -- the latter is the GitHub org and 404s on Hub). Same env vars and still honors `com.centurylinklabs.watchtower.enable`. A `docker login` on the nodes would raise the limit further. **cadvisor no longer publishes to gcr.io** after v0.55 -- the image is now `ghcr.io/google/cadvisor`.
 42. **node-exporter had been launched from `/tmp` (compose project `tmp`) on all 3 nodes**, so `docker compose` in the real dirs didn't own it. Fixed 2026-10-26: removed and redeployed from `/mnt/docker-data/compose/<node>/node-exporter/`. Separately, the **monitoring** compose on .192 still defines a legacy `node-exporter:latest` service on :9100 that a bare `docker compose up -d` would try to start; it is now gated behind `profiles: ["legacy-node-exporter"]`. Also: the matrix-alertmanager-receiver container is now named `monitoring-matrix-alertmanager-receiver-1` (compose default) -- Alertmanager reaches it by service name `matrix-alertmanager-receiver`, which still resolves.
 43. **Farm Manager showed ZERO containers for hf-pdocker-01 (fixed 2026-10-26)**: the server fetches each agent's `/containers` with a timeout and on any error silently returns `[]` for that node. .192's agent took ~12s (47 containers, per-container `stats(stream=False)` = Docker samples twice ~1s apart, only 10 workers) vs a 10s timeout, so the main host's containers simply vanished from the dashboard while `/api/nodes` still said "healthy, 47". Fix: agent uses `stats(..., one_shot=True)` + 20 workers (now ~2s), server timeout 30s. Agent rebuilt on .192 and shipped to .100/bart via `docker save | docker load` (only .192 has the `/opt/farm-manager` source); rollback tags `farm-agent:rollback-20261026` / `farm-manager:rollback-20261026`. Same day: removed a dead `gpu-workstation` node (192.168.86.27) from the live `config.json`, made `/health` ignore Dead containers (bart reported 30 instead of 10), and rebuilt `groups.json` -- every group still used Swarm-era / decommissioned names (`ai-agents_margaret`, `jellyseerr`, `adguardhome`, bart Onyx, `openvas`, `unifi`...), so all group actions were no-ops. Now 14 groups / 63 containers; per-node utilities (farm-agent, node-exporter, telegraf, watchtower) are deliberately ungrouped so a group "stop" can't kill the agents. **4 tests in `tests/test_agent_docker_ops.py` fail -- pre-existing** (they mock `containers.list` but the code uses low-level `api.containers`); not caused by this change.
@@ -1,11 +1,10 @@
 services:
  jellyfin:
-    # PINNED to the 12.0.0 unstable build that migrated the DB to the v12
+    # Pinned to official stable 12.2. History: ran the 2026060106 unstable
-    # schema (DropExtraIdsColumn) — :latest is still 10.11.x and CANNOT read
+    # build from 2026-07-14 because the DB had been migrated to the v12 schema
-    # this DB ("no such column: b.ExtraIds"). Do not unpin until official
+    # and 10.11.x could not read it. Stable 12 shipped 2026-09; moved to
-    # stable >= 12 ships. (2026-07-14: a :latest deploy downgraded it and
+    # 12.2 on 2026-10-17, 12.3 on 2026-10-26. Never go back below 12.
-    # broke Resume/home sections; repinned.)
+    image: jellyfin/jellyfin:12.3.20261023-013714
    image: jellyfin/jellyfin:2026060106
    container_name: jellyfin
    restart: unless-stopped
    environment:
@@ -145,14 +144,19 @@ services:
        max-size: "50m"
        max-file: "3"
-  jellyseerr:
+  seerr:
-    image: fallenbagel/jellyseerr:latest
+    # Seerr = merged successor of Jellyseerr/Overseerr (fallenbagel/jellyseerr
-    container_name: jellyseerr
+    # is deprecated). Migrated 2026-10-17; DB auto-migrated on first start.
    # Runs as node (UID 1000), needs init. LOCAL disk (SQLite). Old config left
    # at /mnt/docker-data/configs/jellyseerr for rollback.
    image: seerr/seerr:v3.5.1
    init: true
    container_name: seerr
    restart: unless-stopped
    environment:
      - TZ=America/Chicago
    volumes:
-      - /mnt/docker-data/configs/jellyseerr:/app/config
+      - /var/lib/seerr:/app/config
    ports:
      - "5055:5055"
    mem_limit: 512M
@@ -1,6 +1,6 @@
 services:
  node-exporter:
-    image: prom/node-exporter:v1.7.0
+    image: prom/node-exporter:v1.12.1
    container_name: node-exporter
    hostname: bart
    network_mode: host
@@ -11,14 +11,14 @@ services:
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
-      - '--path.procfs=/host/proc'
+      - "--path.procfs=/host/proc"
-      - '--path.rootfs=/rootfs'
+      - "--path.rootfs=/rootfs"
-      - '--path.sysfs=/host/sys'
+      - "--path.sysfs=/host/sys"
-      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    logging:
-      driver: "json-file"
+      driver: json-file
      options:
-        max-size: "50m"
+        max-size: 50m
        max-file: "3"
    labels:
-      - "com.centurylinklabs.watchtower.enable=false"
+      - com.centurylinklabs.watchtower.enable=false
@@ -1,6 +1,6 @@
 services:
  watchtower:
-    image: containrrr/watchtower:latest
+    image: nickfedor/watchtower:latest  # containrrr/watchtower is archived upstream (2026-10-26)
    container_name: watchtower
    restart: unless-stopped
    volumes:
@@ -8,7 +8,7 @@ services:
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_INCLUDE_RESTARTING=true
-      - WATCHTOWER_POLL_INTERVAL=86400
+      - WATCHTOWER_SCHEDULE=0 0 4,16 * * *  # twice daily, 04:00 + 16:00 local
      - WATCHTOWER_MONITOR_ONLY=false
      - TZ=America/Chicago
      - DOCKER_API_VERSION=1.44
@@ -4,7 +4,11 @@
 # bridged bind mount). friday-voice calls this via `docker exec hermes hermes ...`.
 services:
  gateway:
-    image: nousresearch/hermes-agent:latest
+    # PINNED 2026-08-20. Was `:latest`, which on this repo is the same digest as
    # `main` (a rolling dev build), and with Watchtower disabled nothing ever
    # re-pulled it — the container sat on v2026.7.1 for ~7 weeks / 9 releases.
    # Bump deliberately; rollback image: nousresearch/hermes-agent:rollback-2026.7.1
    image: nousresearch/hermes-agent:v2026.8.18
    container_name: hermes
    restart: unless-stopped
    volumes:
@@ -0,0 +1,62 @@
 # llama.cpp server on .150 (Docker Desktop) — inference backend for hermes/FRIDAY.
 #
 # Replaced ollama as the *serving* engine on 2026-08-21 after a head-to-head
 # benchmark on the same GGUF, same 64k context, same GPU:
 #   ollama (f16 KV):    15.8 tok/s aggregate, 16.5 sustained, 23207 MiB VRAM
 #   llama.cpp (q8_0 KV): 39.7 tok/s aggregate, 39.9 sustained, 19545 MiB VRAM
 # The difference is the quantized KV cache: it drops the footprint from 23.2GB
 # to 18.6GB, so the whole model fits on the 3090 instead of spilling layers to
 # CPU. Ollama exposes no knob for KV quantization.
 #
 # Image is pinned BY DIGEST on purpose — `:server-cuda` is a moving tag, and
 # this homelab has been bitten by tag-rot three times (synapse, gitea, openclaw).
 services:
  llamacpp:
    image: ghcr.io/ggml-org/llama.cpp@sha256:995fe9f237c622ac257905948c3cdde447ab343db22e0686322445b5ec6d3131
    container_name: llamacpp
    restart: unless-stopped
    ports:
      - "8081:8081"
    volumes:
      # Read-only mount of ollama's blob store. NOTE the coupling: this is the
      # GGUF that `qwen3.8` resolves to. C: only had 26GB free so a dedicated
      # copy wasn't practical. If you ever `ollama rm qwen3.8`, this breaks —
      # copy the blob out first.
      - "C:/Users/Jeremy/.ollama/models/blobs:/models:ro"
    command:
      - "-m"
      - "/models/sha256-f5f1dd8920d417aac2718b0bda3403da274301efdd6760b4f0f4b864ff2ad57d"
      - "--alias"
      - "qwen3.8-64k"
      - "-ngl"
      - "99"                      # all 65 layers to GPU
      - "-c"
      - "65536"                   # hermes hard-refuses anything under 64K
      - "--cache-type-k"
      - "q8_0"                    # the whole point — halves KV cache
      - "--cache-type-v"
      - "q8_0"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8081"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    # The image ships its own HEALTHCHECK that cannot run here, so the
    # container reports permanently "unhealthy" while serving fine (cf. litellm,
    # CLAUDE.md known issue #15). Disable it; liveness is the blackbox probe of
    # GET http://192.168.86.150:8081/health from outside.
    healthcheck:
      disable: true
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
@@ -0,0 +1,35 @@
 # Margaret — OpenClaw gateway agent (hf-pdocker-01, :18790)
 #
 # Reconstructed 2026-08-20 from the layer history of margaret-openclaw:latest;
 # the original Dockerfile was never committed anywhere.
 #
 # OPENCLAW_VERSION is pinned on purpose. This image was previously built with
 # `openclaw@latest`, which is how it sat on 2026.5.27 for months without anyone
 # noticing (same failure mode as the matrixdotorg/synapse :latest rot).
 # Bump this ARG deliberately.
 FROM node:22.22.3-bookworm
 ARG OPENCLAW_VERSION=2026.7.1-2
 RUN apt-get update && apt-get install -y --no-install-recommends \
        git ca-certificates \
    && rm -rf /var/lib/apt/lists/*
 RUN npm install -g openclaw@${OPENCLAW_VERSION} 2>&1 | tail -5
 # The bundled matrix extension ships without its runtime deps installed.
 RUN if [ -d /usr/local/lib/node_modules/openclaw/dist/extensions/matrix ]; then \
      cd /usr/local/lib/node_modules/openclaw/dist/extensions/matrix && \
      npm install --no-fund --no-audit 2>&1 | tail -5; \
    fi
 # Shim for a plugin-sdk export path some builds omit.
 RUN if [ ! -f /usr/local/lib/node_modules/openclaw/dist/plugin-sdk/keyed-async-queue.js ]; then \
      printf 'export { KeyedAsyncQueue, enqueueKeyedTask } from "./index.js";\n' \
        > /usr/local/lib/node_modules/openclaw/dist/plugin-sdk/keyed-async-queue.js; \
    fi
 USER node
 WORKDIR /home/node
 EXPOSE 18790
 CMD ["openclaw", "gateway", "start", "--foreground"]
@@ -0,0 +1,38 @@
 # Spirit — OpenClaw infra-ops agent (hf-pdocker-01, :18795)
 #
 # Reconstructed 2026-08-20 from the layer history of spirit-ops:latest;
 # the original Dockerfile was never committed anywhere.
 #
 # OPENCLAW_VERSION is pinned on purpose. This image was previously built with
 # `openclaw@latest`, which is how it sat on 2026.5.27 for months without anyone
 # noticing (same failure mode as the matrixdotorg/synapse :latest rot).
 # Bump this ARG deliberately.
 FROM node:22.22.3-bookworm
 ARG OPENCLAW_VERSION=2026.7.1-2
 # spirit does infra ops: needs ssh out to the nodes plus basic diagnostics.
 RUN apt-get update && apt-get install -y --no-install-recommends \
        git ca-certificates openssh-client curl jq procps net-tools \
    && rm -rf /var/lib/apt/lists/*
 RUN npm install -g openclaw@${OPENCLAW_VERSION} 2>&1 | tail -5
 RUN npm install -g @google/gemini-cli 2>&1 | tail -3
 # The bundled matrix extension ships without its runtime deps installed.
 RUN if [ -d /usr/local/lib/node_modules/openclaw/dist/extensions/matrix ]; then \
      cd /usr/local/lib/node_modules/openclaw/dist/extensions/matrix && \
      npm install --no-fund --no-audit 2>&1 | tail -5; \
    fi
 # Shim for a plugin-sdk export path some builds omit.
 RUN if [ ! -f /usr/local/lib/node_modules/openclaw/dist/plugin-sdk/keyed-async-queue.js ]; then \
      printf 'export { KeyedAsyncQueue, enqueueKeyedTask } from "./index.js";\n' \
        > /usr/local/lib/node_modules/openclaw/dist/plugin-sdk/keyed-async-queue.js; \
    fi
 USER node
 WORKDIR /home/node
 EXPOSE 18795
 CMD ["openclaw", "gateway", "start", "--foreground"]
@@ -0,0 +1,67 @@
 services:
  mas-db:
    image: postgres:16-alpine
    container_name: mas-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=mas
      - POSTGRES_PASSWORD=3XRBtczkPrdbNgdJftKGWu7QwhT3
      - POSTGRES_DB=mas
    volumes:
      # LOCAL disk, never NFS (Known Issue #16)
      - /var/lib/mas-db:/var/lib/postgresql/data
    networks:
      - mas
    mem_limit: 256M
    memswap_limit: 256M
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U mas -d mas"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
  mas:
    # NOTE: MAS tags are NOT v-prefixed (1.23.0), unlike Synapse (v1.159.0).
    # Distroless image - no shell, so no container healthcheck is possible.
    # Liveness is covered by the blackbox probe of https://auth.hobbs.farm/health.
    image: ghcr.io/element-hq/matrix-authentication-service:1.27.1
    container_name: mas
    restart: unless-stopped
    command: ["server", "--config", "/config/config.yaml"]
    volumes:
      - /mnt/docker-data/configs/mas:/config:ro
      # read-only synapse data dir: syn2mas reads homeserver.yaml from here
      - /var/lib/synapse:/synapse:ro
    ports:
      - "8020:8080"   # web: oauth / compat / discovery / human UI
      - "8021:8081"   # internal: /health only
    networks:
      - mas               # to mas-db
      - matrix-internal   # to synapse (admin API) and back
    depends_on:
      mas-db:
        condition: service_healthy
    mem_limit: 256M
    memswap_limit: 256M
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
 networks:
  mas:
    driver: bridge
  matrix-internal:
    external: true
    name: matrix_matrix-internal
@@ -1,20 +1,20 @@
 services:
  synapse:
-    image: matrixdotorg/synapse:latest
+    # 2026-08-20: matrixdotorg repo is abandoned (froze at 1.148.0); upstream is now element-hq.
    # Pinned deliberately so the running version is visible in the file - bump manually.
    image: ghcr.io/element-hq/synapse:v1.163.2
    container_name: synapse
    restart: unless-stopped
    environment:
      - SYNAPSE_SERVER_NAME=mess.hobbs.farm
      - SYNAPSE_REPORT_STATS=no
    volumes:
-      - /mnt/docker-data/configs/matrix/data:/data
+      # 2026-08-20: moved off NFS -> local NVMe (SQLite locking, Known Issue #16)
      - /var/lib/synapse:/data
    ports:
      - "8008:8008"
    networks:
      - matrix-internal
    depends_on:
      synapse-db:
        condition: service_healthy
    mem_limit: 1G
    memswap_limit: 1G
    healthcheck:
@@ -52,21 +52,22 @@ services:
        max-file: "3"
  synapse-db:
-    image: postgres:15-alpine
+    image: postgres:16-alpine
    container_name: synapse-db
    user: "1024:100"
    restart: unless-stopped
    environment:
      - POSTGRES_USER=synapse
-      - POSTGRES_PASSWORD=synapse
+      - POSTGRES_PASSWORD=MnLASftPXou9ubOmawmALYwROwgM
      - POSTGRES_DB=synapse
      # C collation is MANDATORY - Synapse refuses any other
      - POSTGRES_INITDB_ARGS=--encoding=UTF-8 --lc-collate=C --lc-ctype=C
    volumes:
-      - /mnt/docker-data/databases/matrix-db:/var/lib/postgresql/data
+      # LOCAL disk, never NFS (Known Issue #16). No user: 1024:100 - that is an NFS workaround.
      - /var/lib/synapse-db:/var/lib/postgresql/data
    networks:
      - matrix-internal
-    mem_limit: 256M
+    mem_limit: 512M
-    memswap_limit: 256M
+    memswap_limit: 512M
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    healthcheck:
@@ -1,21 +1,12 @@
 networks:
  monitoring-net:
    name: monitoring-net
 services:
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: unless-stopped
    user: "104"
    networks:
      - monitoring-net
    ports:
      - "3001:3000"
    volumes:
      - /mnt/docker-data/configs/grafana:/var/lib/grafana
    mem_limit: 512M
    memswap_limit: 512M
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://localhost:3000/api/health || exit 1"]
      interval: 30s
@@ -27,13 +18,19 @@ services:
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == hf-pdocker-01
      resources:
        limits:
          memory: 512M
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: unless-stopped
    networks:
      - monitoring-net
    ports:
      - "9095:9090"
    command:
@@ -44,8 +41,6 @@ services:
      - '--web.enable-lifecycle'
    volumes:
      - /mnt/docker-data/configs/prometheus:/etc/prometheus
    mem_limit: 512M
    memswap_limit: 512M
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://localhost:9090/-/healthy || exit 1"]
      interval: 30s
@@ -57,13 +52,19 @@ services:
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == hf-pdocker-01
      resources:
        limits:
          memory: 512M
  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    restart: unless-stopped
    networks:
      - monitoring-net
    ports:
      - "9093:9093"
    command:
@@ -72,8 +73,6 @@ services:
    volumes:
      - /mnt/docker-data/configs/alertmanager:/etc/alertmanager
      - alertmanager-data:/alertmanager
    mem_limit: 256M
    memswap_limit: 256M
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://localhost:9093/-/healthy || exit 1"]
      interval: 30s
@@ -85,19 +84,49 @@ services:
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == hf-pdocker-01
      resources:
        limits:
          memory: 256M
  node-exporter:
    # DISABLED 2026-10-26: node-exporter runs STANDALONE (compose/<node>/node-exporter, pid:host).
    # This legacy copy would collide on :9100 if started; gated behind a profile so a
    # bare `docker compose up -d` here never starts it.
    profiles: ["legacy-node-exporter"]
    image: prom/node-exporter:latest
    restart: unless-stopped
    ports:
      - "9100:9100"
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: global
      resources:
        limits:
          memory: 128M
  influxdb:
    image: influxdb:1.8-alpine
    container_name: influxdb
    restart: unless-stopped
    networks:
      - monitoring-net
    ports:
      - "8086:8086"
    volumes:
      - /mnt/docker-data/configs/influxdb:/var/lib/influxdb
    mem_limit: 1G
    memswap_limit: 1G
    healthcheck:
      test: ["CMD-SHELL", "wget -q --spider http://localhost:8086/ping || exit 1"]
      interval: 30s
@@ -109,45 +138,99 @@ services:
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == hf-pdocker-01
      resources:
        limits:
          memory: 1G
  matrix-alertmanager-receiver:
-    image: metio/matrix-alertmanager-receiver:2026.2.25
+    image: metio/matrix-alertmanager-receiver:2026.9.23
    container_name: matrix-alertmanager-receiver
    restart: unless-stopped
    networks:
      - monitoring-net
    ports:
      - "9094:9094"
    volumes:
      - /mnt/docker-data/configs/alertmanager/matrix-receiver-config.yml:/config/config.yml:ro
    command: ["--config-path", "/config/config.yml"]
-    mem_limit: 128M
+    # NOTE: No healthcheck - this image is distroless (no shell), so CMD-SHELL healthchecks fail.
-    memswap_limit: 128M
+    # The service is verified healthy by Alertmanager successfully POSTing to it.
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == hf-pdocker-01
      resources:
        limits:
          memory: 128M
  cadvisor:
    image: ghcr.io/google/cadvisor:v0.60.6  # moved off gcr.io (no releases there after v0.55)
    restart: unless-stopped
    privileged: true
    command:
      - '--docker_only=true'
      - '--disable_metrics=advtcp,cpu_topology,cpuset,hugetlb,memory_numa,process,referenced_memory,resctrl,sched,tcp,udp,disk,diskIO,percpu'
      - '--housekeeping_interval=30s'
      - '--store_container_labels=false'
    ports:
      - "8081:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    devices:
      - /dev/kmsg
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == hf-pdocker-01
      resources:
        limits:
          memory: 256M
  blackbox-exporter:
    image: prom/blackbox-exporter:latest
    container_name: blackbox-exporter
    restart: unless-stopped
    networks:
      - monitoring-net
    ports:
      - "9115:9115"
    command:
      - '--config.file=/config/blackbox.yml'
    volumes:
      - /mnt/docker-data/configs/blackbox:/config:ro
    mem_limit: 128M
    memswap_limit: 128M
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.hostname == hf-pdocker-01
      resources:
        limits:
          memory: 128M
 volumes:
  alertmanager-data:
@@ -1,6 +1,6 @@
 services:
  node-exporter:
-    image: prom/node-exporter:v1.7.0
+    image: prom/node-exporter:v1.12.1
    container_name: node-exporter
    hostname: hf-pdocker-01
    network_mode: host
@@ -0,0 +1,102 @@
 http:
  routers:
    auth:
      rule: "Host(`auth.hobbs.farm`)"
      service: mas
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      tls:
        certResolver: letsencrypt
    # 2026-08-20: under MAS, Synapse no longer serves the login endpoints
    # (it returns M_UNRECOGNIZED). These three compat paths must reach MAS or
    # no client can sign in. Explicit high priority so it wins over `mess`.
    mess-auth-compat:
      rule: "Host(`mess.hobbs.farm`) && PathRegexp(`^/_matrix/client/(r0|v3|v1|unstable)/(login|logout|refresh)`)"
      priority: 1000
      service: mas
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      tls:
        certResolver: letsencrypt
    yoda:
      rule: "Host(`yoda.hobbs.farm`)"
      service: yoda
      entryPoints:
        - websecure
      middlewares:
        - security-headers-allow-frames
      tls:
        certResolver: letsencrypt
    mess:
      rule: "Host(`mess.hobbs.farm`)"
      service: mess
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      tls:
        certResolver: letsencrypt
    mess-federation:
      rule: "Host(`mess.hobbs.farm`) && PathPrefix(`/_matrix/federation`)"
      service: mess
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      tls:
        certResolver: letsencrypt
    wellknown:
      rule: "Host(`hobbs.farm`) && PathPrefix(`/.well-known/matrix`)"
      service: wellknown
      entryPoints:
        - websecure
      middlewares:
        - security-headers
      tls:
        certResolver: letsencrypt
    solo:
      rule: "Host(`solo.hobbs.farm`)"
      service: solo
      entryPoints:
        - websecure
      middlewares:
        - solo-basicauth
        - security-headers-allow-frames
      tls:
        certResolver: letsencrypt
  services:
    mas:
      loadBalancer:
        servers:
          - url: "http://192.168.86.192:8020"
    yoda:
      loadBalancer:
        servers:
          - url: "http://192.168.86.192:8123"
    mess:
      loadBalancer:
        servers:
          - url: "http://192.168.86.192:8008"
    wellknown:
      loadBalancer:
        servers:
          - url: "http://192.168.86.192:8010"
    solo:
      loadBalancer:
        servers:
          - url: "http://192.168.86.192:8199"
@@ -1,6 +1,6 @@
 services:
  traefik:
-    image: traefik:v3.6
+    image: traefik:v3.7
    container_name: traefik
    restart: unless-stopped
    command:
@@ -1,6 +1,6 @@
 services:
  gitea:
-    image: gitea/gitea:1.26.1
+    image: gitea/gitea:1.26.4
    container_name: gitea
    restart: unless-stopped
    labels:
@@ -1,6 +1,6 @@
 services:
  node-exporter:
-    image: prom/node-exporter:v1.7.0
+    image: prom/node-exporter:v1.12.1
    container_name: node-exporter
    hostname: hf-pdocker-02
    network_mode: host
@@ -11,14 +11,14 @@ services:
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
-      - '--path.procfs=/host/proc'
+      - "--path.procfs=/host/proc"
-      - '--path.rootfs=/rootfs'
+      - "--path.rootfs=/rootfs"
-      - '--path.sysfs=/host/sys'
+      - "--path.sysfs=/host/sys"
-      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
+      - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)"
    logging:
-      driver: "json-file"
+      driver: json-file
      options:
-        max-size: "50m"
+        max-size: 50m
        max-file: "3"
    labels:
-      - "com.centurylinklabs.watchtower.enable=false"
+      - com.centurylinklabs.watchtower.enable=false
@@ -0,0 +1,3 @@
 Decommissioned 2026-10-05 and again 2026-10-26 (it had been revived and was running).
 Duplicate controller; the real UniFi controller is the Podman UniFi OS Server on .150.
 Do NOT compose up. Data kept in /var/lib/unifi-os (.100) + /mnt/docker-data/configs/unifi-os/persistent.
@@ -1,6 +1,6 @@
 services:
  watchtower:
-    image: containrrr/watchtower:latest
+    image: nickfedor/watchtower:latest  # containrrr/watchtower is archived upstream (2026-10-26)
    container_name: watchtower
    restart: unless-stopped
    volumes:
@@ -8,12 +8,11 @@ services:
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_INCLUDE_RESTARTING=true
-      - WATCHTOWER_POLL_INTERVAL=86400
+      - WATCHTOWER_SCHEDULE=0 0 4,16 * * *  # twice daily, 04:00 + 16:00 local
      - WATCHTOWER_MONITOR_ONLY=false
      - TZ=America/Chicago
      - DOCKER_API_VERSION=1.44
    mem_limit: 256M
    memswap_limit: 256M
    logging:
      driver: "json-file"
      options:
@@ -0,0 +1,668 @@
 # QR Sign-In via Matrix Authentication Service — Implementation Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Enable "sign in with QR code" (MSC4108) for Matrix accounts on hobbs.farm, which requires deploying Matrix Authentication Service (MAS) as the homeserver's auth provider.
 **Architecture:** QR sign-in is an OAuth 2.0 device grant, not a password login. Synapse 1.159.0 refuses to start with `msc4108_enabled: true` unless `matrix_authentication_service.enabled` is also true. MAS requires PostgreSQL, and — critically — **`syn2mas` can only read a Synapse whose own database is PostgreSQL**. Synapse here is on SQLite, so this is a three-stage chain: port Synapse to PostgreSQL, deploy MAS and migrate the accounts into it, then turn on the QR MSCs.
 **Tech Stack:** Synapse 1.159.0 (SQLite → PostgreSQL 16), Matrix Authentication Service (Docker, ghcr.io/element-hq), PostgreSQL 16-alpine ×2, Traefik, Cloudflare DNS, Docker Compose on hf-pdocker-01.
 ## Why this is three migrations, not a config flag
 Verified in the running image's own source (`synapse/config/experimental.py`):
 ```python
 if (self.msc4108_enabled or self.msc4108_delegation_endpoint is not None) and not auth_delegated:
    raise ConfigError("MSC4108 requires matrix_authentication_service to be enabled", ...)
 ```
 The same guard exists for MSC4388 (the QR out-of-band channel). And from the MAS docs:
 > "It is worth noting that MAS currently only supports PostgreSQL as a database backend."
 > "The migration tool only supports reading from PostgreSQL for the Synapse database as well."
 Synapse on this node currently runs `database: {name: sqlite3, database: /data/homeserver.db}`.
 ## Global Constraints
 - Synapse image stays pinned at `ghcr.io/element-hq/synapse:v1.159.0` (**`v` prefix required**; `:1.159.0` 404s). Do not bump during this work.
 - Synapse ≥1.143 requires **PostgreSQL 14+**. Use `postgres:16-alpine`.
 - Synapse's PostgreSQL database MUST be created with `ENCODING 'UTF8' LC_COLLATE 'C' LC_CTYPE 'C'` — Synapse refuses any other collation.
 - MAS requires its **own separate, blank** PostgreSQL database. Do not share Synapse's.
 - All lock-prone databases go on **LOCAL disk**, never NFS (CLAUDE.md Known Issue #16). Use `/var/lib/synapse-db` and `/var/lib/mas-db`.
 - On hf-pdocker-01, **`memswap_limit` must equal `mem_limit`** on every container (Known Issue #12 — a drift here crashed the whole node on 2026-07-12).
 - Node headroom: ~8GB available of 23GB, Frigate is the main consumer. Budget ≤600MB total for the new containers.
 - **hobbs.farm has no wildcard DNS record** (memory `hobbs-farm-dns`). Traefik uses DNS-01, so a certificate will be issued for a hostname with **no A record** — a valid cert does NOT mean the name resolves. Any new public hostname needs an explicit Cloudflare A record → `98.160.103.126`, DNS-only (grey cloud).
 - Watchtower does not run on .192; pin every image tag explicitly.
 - Synapse data lives at `/var/lib/synapse` on local NVMe. The NFS copy at `/mnt/docker-data/configs/matrix/data` is a stale backup — **never remount it**.
 - The mandatory post-change checklist in CLAUDE.md applies (CLAUDE.md, MEMORY.md, Prometheus probes, wiki, Farm Manager links).
 ## Blast Radius
 Authentication for **all 19 accounts** moves to MAS. Known consumers that must keep working:
 | Account | How it authenticates | Migrates via |
 |---|---|---|
 | `@jeremy` | password `bigdog01` + 5 Element devices (2× Element X Android, 2× Element Desktop) | password hash + devices |
 | `@frigate` | long-lived access token `syt_ZnJpZ2F0…` in `frigate-notify/.env` | access token |
 | `@alertbot` | long-lived access token `syt_YWxlcnRib3Q…` in `matrix-receiver-config.yml` | access token |
 | `@margaret` | openclaw.json contains **both** `accessToken` and `password` | both |
 | `@spirit` | openclaw config (path differs from margaret's — resolve in Task 1) | TBD in Task 1 |
 | `@clippyauto` | n8n credential on .100, polls `/sync` every ~2s | TBD in Task 1 |
 | `@fry`, `@hermes`, `@julie`, `@pluto`, `@testbot`, `@admin1`, `@adminuser` | unknown | TBD in Task 1 |
 MAS exposes a compatibility layer proxying `/_matrix/client/*/login`, so password-based bots are expected to keep working — **this is the single biggest assumption in the plan and Task 1 exists to test it, not trust it.**
 ## File Structure
 - Create `compose/hf-pdocker-01/mas/docker-compose.yml` — MAS + `mas-db` PostgreSQL. One file, one responsibility: authentication.
 - Modify `compose/hf-pdocker-01/matrix/docker-compose.yml` — re-add a `synapse-db` service (PostgreSQL 16, local disk). Note this partly reverses the 2026-08-20 removal of the *empty* postgres; this one will actually hold data.
 - Create `/var/lib/synapse/homeserver-postgres.yaml` on the node — port-target config for `synapse_port_db` (node-only, not in repo — contains DB credentials).
 - Create `/mnt/docker-data/configs/mas/config.yaml` on the node — MAS config (node-only, contains signing keys and secrets; back up to Vaultwarden).
 - Modify `/mnt/docker-data/traefik/dynamic/temp-routes.yml` — add the `auth.hobbs.farm` router + service. (Yes, the Matrix, Home Assistant and Guacamole routes all live in a file called `temp-routes.yml`. It is load-bearing despite the name.)
 - Modify `CLAUDE.md`, `MEMORY.md` — post-change checklist.
 ---
 ### Task 1: Auth inventory and full backup
 **Files:**
 - Create: `/var/lib/synapse.bak-premas/` (node), `docs/superpowers/plans/mas-auth-inventory.md` (repo)
 **Interfaces:**
 - Produces: a complete table of every account → auth method → config file location, used by Tasks 7 and 10 to verify nothing broke.
 - [ ] **Step 1: Enumerate every device and its last-seen IP, to know what "still working" means later**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec synapse python3 -c "
 import sqlite3
 c=sqlite3.connect(\"file:/var/lib/synapse/homeserver.db?mode=ro\",uri=True)
 for u,d,n,ip in c.execute(\"select user_id,device_id,display_name,last_seen_ip from devices order by user_id\"):
    print(u,d,n,ip)
 "'
 ```
 Expected: rows for `@jeremy` (5 devices), plus bot devices for margaret/spirit/clippyauto/frigate/alertbot/fry.
 - [ ] **Step 2: Locate spirit's and clippyauto's credentials (unresolved in planning)**
 ```bash
 ssh glyptic@192.168.86.192 'docker inspect spirit --format "{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}"; docker exec spirit sh -c "ls -la /home/glyptic/.openclaw/ 2>/dev/null || find / -name openclaw.json -maxdepth 6 2>/dev/null | head"'
 ssh glyptic@192.168.86.100 'docker exec n8n sh -c "printenv | grep -i matrix" 2>/dev/null; echo "---"; docker exec n8n-db psql -U n8n -tc "select name, type from credentials_entity;" 2>/dev/null | head'
 ```
 Expected: a config path for spirit and a credential type for n8n. Record whether each uses an **access token** or a **password** — access tokens migrate untouched; passwords depend on the MAS compat layer.
 - [ ] **Step 3: Write the inventory file**
 Record one row per account: user id, auth method, exact file path holding the credential, and how to verify it post-migration. Save as `docs/superpowers/plans/mas-auth-inventory.md`.
 - [ ] **Step 4: Take a cold, complete backup**
 ```bash
 ssh glyptic@192.168.86.192 '
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose stop synapse
 sudo rsync -a /var/lib/synapse/ /var/lib/synapse.bak-premas/
 sudo python3 -c "
 import sqlite3;c=sqlite3.connect(\"/var/lib/synapse.bak-premas/homeserver.db\")
 print(\"integrity:\", c.execute(\"pragma integrity_check\").fetchone()[0])
 print(\"users:\", c.execute(\"select count(*) from users\").fetchone()[0])
 print(\"events:\", c.execute(\"select count(*) from events\").fetchone()[0])
 "
 docker compose start synapse'
 ```
 Expected: `integrity: ok`, 19 users, ~93,884 events. **Do not proceed if integrity is anything but `ok`.**
 - [ ] **Step 5: Commit the inventory**
 ```bash
 git add docs/superpowers/plans/mas-auth-inventory.md
 git commit -m "docs: matrix auth inventory ahead of MAS migration"
 ```
 ---
 ### Task 2: PostgreSQL for Synapse (blank, correct collation)
 **Files:**
 - Modify: `compose/hf-pdocker-01/matrix/docker-compose.yml`
 **Interfaces:**
 - Produces: `synapse-db` reachable at `synapse-db:5432` on the `matrix-internal` network, database `synapse`, user `synapse`, C collation. Task 3 ports data into it.
 - [ ] **Step 1: Add the service to the compose file**
 ```yaml
  synapse-db:
    image: postgres:16-alpine
    container_name: synapse-db
    restart: unless-stopped
    environment:
      - POSTGRES_USER=synapse
      - POSTGRES_PASSWORD=<generate: openssl rand -base64 24>
      - POSTGRES_DB=synapse
      - POSTGRES_INITDB_ARGS=--encoding=UTF-8 --lc-collate=C --lc-ctype=C
    volumes:
      - /var/lib/synapse-db:/var/lib/postgresql/data   # LOCAL disk - Known Issue #16
    networks:
      - matrix-internal
    mem_limit: 512M
    memswap_limit: 512M          # MUST equal mem_limit - Known Issue #12
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U synapse -d synapse"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    logging:
      driver: "json-file"
      options: {max-size: "50m", max-file: "3"}
 ```
 Note: **no `user: "1024:100"`** — that workaround is only for NFS root_squash. This volume is local.
 - [ ] **Step 2: Deploy and verify collation**
 ```bash
 scp compose/hf-pdocker-01/matrix/docker-compose.yml glyptic@192.168.86.192:/mnt/docker-data/compose/hf-pdocker-01/matrix/
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose up -d synapse-db && sleep 20 && docker exec synapse-db psql -U synapse -d synapse -c "show lc_collate;"'
 ```
 Expected: `lc_collate` = `C`. **If it says anything else, delete `/var/lib/synapse-db` and redo** — Synapse will refuse to start otherwise, and fixing it later means re-porting.
 - [ ] **Step 3: Commit**
 ```bash
 git add compose/hf-pdocker-01/matrix/docker-compose.yml
 git commit -m "feat: add postgres for synapse (local disk, C collation) ahead of MAS"
 ```
 ---
 ### Task 3: Port Synapse SQLite → PostgreSQL
 **Files:**
 - Create: `/var/lib/synapse/homeserver-postgres.yaml` (node only)
 - Modify: `/var/lib/synapse/homeserver.yaml` (node only)
 **Interfaces:**
 - Consumes: `synapse-db` from Task 2.
 - Produces: a Synapse running on PostgreSQL — the precondition `syn2mas` requires in Task 7.
 - [ ] **Step 1: Write the port-target config**
 Copy `/var/lib/synapse/homeserver.yaml` to `homeserver-postgres.yaml` and replace the `database:` block with:
 ```yaml
 database:
  name: psycopg2
  args:
    user: synapse
    password: <the password from Task 2>
    dbname: synapse
    host: synapse-db
    cp_min: 5
    cp_max: 10
 ```
 - [ ] **Step 2: Stop Synapse (downtime starts)**
 ```bash
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose stop synapse'
 ```
 - [ ] **Step 3: Run the port**
 ```bash
 ssh glyptic@192.168.86.192 'docker run --rm --network matrix_matrix-internal \
  -v /var/lib/synapse:/data ghcr.io/element-hq/synapse:v1.159.0 \
  synapse_port_db --sqlite-database /data/homeserver.db --postgres-config /data/homeserver-postgres.yaml'
 ```
 Expected: a progress table per-table, ending in `Success!`. Budget 10–40 minutes for ~94k events on this hardware. If it fails partway, drop and recreate the database before retrying — **the port tool is not idempotent against a partially-filled database.**
 - [ ] **Step 4: Verify row counts match the source**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec synapse-db psql -U synapse -d synapse -tc "select (select count(*) from users) as users, (select count(*) from events) as events;"'
 ```
 Expected: `19 | 93884` (or higher for events if the server ran between backup and port). **Must not be lower.**
 - [ ] **Step 5: Cut Synapse over**
 ```bash
 ssh glyptic@192.168.86.192 'sudo cp /var/lib/synapse/homeserver.yaml /var/lib/synapse/homeserver.yaml.bak-presql
 sudo cp /var/lib/synapse/homeserver-postgres.yaml /var/lib/synapse/homeserver.yaml
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose start synapse'
 ```
 - [ ] **Step 6: Verify the service on PostgreSQL**
 ```bash
 ssh glyptic@192.168.86.192 '
 for i in $(seq 1 30); do [ "$(docker inspect synapse --format "{{.State.Health.Status}}")" = healthy ] && echo healthy && break; sleep 10; done
 curl -s http://localhost:8008/_synapse/admin/v1/server_version; echo
 curl -s -X POST http://localhost:8008/_matrix/client/v3/login -H "Content-Type: application/json" \
  -d "{\"type\":\"m.login.password\",\"identifier\":{\"type\":\"m.id.user\",\"user\":\"jeremy\"},\"password\":\"bigdog01\"}" | head -c 80
 docker logs synapse --since 5m 2>&1 | grep -cE " ERROR "'
 ```
 Expected: `healthy`, version `1.159.0`, an `access_token` in the login response, `0` errors. Log out that verification session afterwards (`POST /_matrix/client/v3/logout`).
 - [ ] **Step 7: Confirm bots reconnected before declaring the task done**
 ```bash
 ssh glyptic@192.168.86.192 'sleep 60; docker logs synapse --since 60s 2>&1 | grep -oE "\{@[a-z0-9]+:hobbs.farm\}" | sort | uniq -c'
 ```
 Expected: clippyauto, margaret, spirit, fry all present. **Rollback if not:** restore `homeserver.yaml.bak-presql` and restart — SQLite file is untouched and still authoritative until Task 7.
 - [ ] **Step 8: Commit**
 ```bash
 git commit --allow-empty -m "chore: synapse ported from sqlite to postgres (node config, not in repo)"
 ```
 ---
 ### Task 4: PostgreSQL for MAS
 **Files:**
 - Create: `compose/hf-pdocker-01/mas/docker-compose.yml`
 **Interfaces:**
 - Produces: `mas-db:5432`, database `mas`, user `mas`, on a new `mas` bridge network that also carries the MAS container.
 - [ ] **Step 1: Create the compose file with the database service only**
 Same shape as Task 2 Step 1, but: `container_name: mas-db`, `POSTGRES_USER=mas`, `POSTGRES_DB=mas`, volume `/var/lib/mas-db:/var/lib/postgresql/data`, `mem_limit: 256M`, `memswap_limit: 256M`, network `mas`. Default collation is fine — the C-collation requirement is Synapse's, not MAS's.
 - [ ] **Step 2: Deploy and verify**
 ```bash
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/mas && docker compose up -d mas-db && sleep 20 && docker exec mas-db psql -U mas -d mas -tc "select 1;"'
 ```
 Expected: `1`.
 - [ ] **Step 3: Commit**
 ```bash
 git add compose/hf-pdocker-01/mas/docker-compose.yml
 git commit -m "feat: add mas-db postgres for matrix authentication service"
 ```
 ---
 ### Task 5: Generate MAS config and deploy the service
 **Files:**
 - Create: `/mnt/docker-data/configs/mas/config.yaml` (node only — contains secrets)
 - Modify: `compose/hf-pdocker-01/mas/docker-compose.yml`
 **Interfaces:**
 - Consumes: `mas-db` from Task 4.
 - Produces: MAS listening on host port **8020** (verified free on .192: 8008/8010/8080/8082/8085/8086/8090/8123/8181/8199/8300/8310 are taken).
 - [ ] **Step 1: Generate a config seeded from the Synapse config**
 ```bash
 ssh glyptic@192.168.86.192 'sudo mkdir -p /mnt/docker-data/configs/mas && docker run --rm \
  -v /var/lib/synapse:/synapse:ro -v /mnt/docker-data/configs/mas:/out \
  ghcr.io/element-hq/matrix-authentication-service:latest \
  config generate --synapse-config /synapse/homeserver.yaml --output /out/config.yaml'
 ```
 Expected: `config.yaml` written, containing generated signing keys and a `matrix.secret` shared secret.
 - [ ] **Step 2: Set the database URI, public base, and password hashing scheme**
 Edit `/mnt/docker-data/configs/mas/config.yaml`:
 ```yaml
 database:
  uri: postgresql://mas:<password>@mas-db/mas
 http:
  public_base: https://auth.hobbs.farm
  listeners:
    - name: web
      resources: [{name: discovery}, {name: human}, {name: oauth}, {name: compat}, {name: graphql}, {name: assets}]
      binds: [{address: "[::]:8080"}]
 passwords:
  enabled: true
  schemes:
    - version: 1
      algorithm: bcrypt
      unicode_normalization: true    # REQUIRED for imported Synapse hashes
    - version: 2
      algorithm: argon2id
 ```
 The `unicode_normalization: true` on the bcrypt v1 scheme is what makes migrated Synapse password hashes verify. Without it, **every migrated password silently fails**.
 - [ ] **Step 3: Add the MAS service to compose**
 ```yaml
  mas:
    image: ghcr.io/element-hq/matrix-authentication-service:latest   # pin to the digest after first pull
    container_name: mas
    restart: unless-stopped
    command: ["server", "--config", "/config/config.yaml"]
    volumes:
      - /mnt/docker-data/configs/mas:/config:ro
    ports:
      - "8020:8080"
    networks: [mas, matrix-internal]
    depends_on:
      mas-db: {condition: service_healthy}
    mem_limit: 256M
    memswap_limit: 256M
    labels:
      - "com.centurylinklabs.watchtower.enable=false"
 ```
 MAS needs to reach Synapse's admin API and Synapse needs to reach MAS, hence both networks.
 - [ ] **Step 4: Run migrations and start**
 ```bash
 ssh glyptic@192.168.86.192 'cd /mnt/docker-data/compose/hf-pdocker-01/mas && docker compose run --rm mas database migrate --config /config/config.yaml && docker compose up -d mas && sleep 15 && curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8020/health'
 ```
 Expected: migrations apply cleanly, health endpoint returns `200`.
 - [ ] **Step 5: Store the MAS secrets in Vaultwarden and commit**
 The generated `config.yaml` holds signing keys that cannot be regenerated without invalidating sessions. Store a copy in Vaultwarden (note: the vault master password is currently unknown — see memory `vault-master-password-stale`; resolve that first or keep an encrypted copy on the NAS).
 ```bash
 git add compose/hf-pdocker-01/mas/docker-compose.yml
 git commit -m "feat: deploy matrix authentication service on :8020"
 ```
 ---
 ### Task 6: syn2mas check and dry-run
 **Interfaces:**
 - Consumes: Synapse-on-PostgreSQL (Task 3), MAS + DB (Tasks 4–5).
 - Produces: a clean dry-run, the go/no-go gate for Task 7.
 - [ ] **Step 1: Run the checker**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec mas mas-cli syn2mas check --config /config/config.yaml --synapse-config /synapse/homeserver.yaml'
 ```
 (Mount `/var/lib/synapse` read-only into the MAS container as `/synapse` for this — add it in Task 5 Step 3 if not already present.)
 Expected: no blocking findings. Known blockers it looks for: custom password provider modules, SAML/LDAP providers. This deployment has neither.
 - [ ] **Step 2: Dry run**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec mas mas-cli syn2mas migrate --config /config/config.yaml --synapse-config /synapse/homeserver.yaml --dry-run'
 ```
 Expected: a summary of users, password hashes, devices, and access tokens it *would* import. Cross-check the user count against **19** and the device count against Task 1 Step 1. **Do not proceed to Task 7 if the counts disagree.**
 - [ ] **Step 3: Record the dry-run output in the inventory doc and commit**
 ```bash
 git add docs/superpowers/plans/mas-auth-inventory.md
 git commit -m "docs: syn2mas dry-run results"
 ```
 ---
 ### Task 7: Migrate accounts into MAS and delegate Synapse auth
 **Files:**
 - Modify: `/var/lib/synapse/homeserver.yaml` (node only)
 **Interfaces:**
 - Produces: Synapse delegating all authentication to MAS — the precondition MSC4108 requires.
 **This is the point of no return.** The MAS docs state migration "is not easily reversible and requires careful backup procedures before proceeding." Rollback means restoring both databases from backup.
 - [ ] **Step 1: Fresh backup of both databases immediately before migrating**
 ```bash
 ssh glyptic@192.168.86.192 '
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose stop synapse
 docker exec synapse-db pg_dump -U synapse synapse | gzip > /tmp/synapse-premas.sql.gz
 docker exec mas-db pg_dump -U mas mas | gzip > /tmp/mas-premigrate.sql.gz
 sudo mv /tmp/*.gz /var/lib/synapse.bak-premas/ && ls -la /var/lib/synapse.bak-premas/*.gz'
 ```
 Expected: two non-empty `.gz` files.
 - [ ] **Step 2: Run the migration (Synapse stays stopped)**
 ```bash
 ssh glyptic@192.168.86.192 'docker exec mas mas-cli syn2mas migrate --config /config/config.yaml --synapse-config /synapse/homeserver.yaml'
 ```
 Expected: completes with an import summary matching the dry-run.
 - [ ] **Step 3: Point Synapse at MAS**
 Add to `/var/lib/synapse/homeserver.yaml`:
 ```yaml
 matrix_authentication_service:
  enabled: true
  endpoint: http://mas:8080/
  secret: <the matrix.secret value from /mnt/docker-data/configs/mas/config.yaml>
 ```
 - [ ] **Step 4: Start Synapse and verify delegation is live**
 ```bash
 ssh glyptic@192.168.86.192 '
 cd /mnt/docker-data/compose/hf-pdocker-01/matrix && docker compose start synapse
 for i in $(seq 1 30); do [ "$(docker inspect synapse --format "{{.State.Health.Status}}")" = healthy ] && echo healthy && break; sleep 10; done
 curl -s -o /dev/null -w "auth_metadata: %{http_code}\n" http://localhost:8008/_matrix/client/unstable/org.matrix.msc2965/auth_metadata'
 ```
 Expected: `healthy`, and auth_metadata now returns **200** (it returns 404 today).
 - [ ] **Step 5: Verify every account from the Task 1 inventory still authenticates**
 ```bash
 ssh glyptic@192.168.86.192 '
 echo "--- password login via MAS compat layer:"
 curl -s -X POST http://localhost:8008/_matrix/client/v3/login -H "Content-Type: application/json" \
  -d "{\"type\":\"m.login.password\",\"identifier\":{\"type\":\"m.id.user\",\"user\":\"jeremy\"},\"password\":\"bigdog01\"}" | head -c 100; echo
 echo "--- migrated access token (frigate):"
 curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8008/_matrix/client/v3/account/whoami -H "Authorization: Bearer syt_ZnJpZ2F0<rest-from-.env>"
 echo "--- bots syncing:"
 sleep 90; docker logs synapse --since 90s 2>&1 | grep -oE "\{@[a-z0-9]+:hobbs.farm\}" | sort | uniq -c'
 ```
 Expected: an `access_token` for jeremy, `200` for the frigate token, and margaret/spirit/clippyauto/fry all syncing. **If password login fails, the bcrypt `unicode_normalization: true` setting from Task 5 Step 2 is the first thing to check.**
 - [ ] **Step 6: Commit**
 ```bash
 git commit --allow-empty -m "feat: synapse auth delegated to MAS (node config)"
 ```
 ---
 ### Task 8: Publish MAS and add DNS
 **Files:**
 - Modify: `/mnt/docker-data/traefik/dynamic/temp-routes.yml` (confirmed: this is the file holding the `mess`, `wellknown`, `yoda` and `solo` routers)
 - External: Cloudflare DNS
 - [ ] **Step 1: Add the Cloudflare A record FIRST**
 `auth.hobbs.farm` → `98.160.103.126`, **DNS-only (grey cloud)**. There is no wildcard record on this zone, and Traefik's DNS-01 challenge will happily issue a valid certificate for a hostname that does not resolve — so a working cert proves nothing here (memory `hobbs-farm-dns`).
 Verify from outside the LAN before continuing:
 ```bash
 getent hosts auth.hobbs.farm
 ```
 Expected: `98.160.103.126`.
 - [ ] **Step 2: Add the Traefik router and service**
 ```yaml
    auth:
      rule: "Host(`auth.hobbs.farm`)"
      service: auth
      entryPoints: [websecure]
      middlewares: [security-headers]
      tls: {certResolver: letsencrypt}
 ```
 ```yaml
    auth:
      loadBalancer:
        servers:
          - url: "http://192.168.86.192:8020"
 ```
 Do **not** add basicAuth — clients must reach this unauthenticated to log in.
 - [ ] **Step 3: Verify publicly**
 ```bash
 curl -s -o /dev/null -w "%{http_code}\n" https://auth.hobbs.farm/health
 curl -s https://mess.hobbs.farm/_matrix/client/unstable/org.matrix.msc2965/auth_metadata | head -c 300
 ```
 Expected: `200`, and auth metadata advertising `https://auth.hobbs.farm` endpoints.
 - [ ] **Step 4: Confirm `.well-known` needs no change (verify, do not assume)**
 Current contents, served from `/mnt/docker-data/configs/matrix/wellknown/` via `matrix-wellknown` on :8010:
 ```
 client -> {"m.homeserver": {"base_url": "https://mess.hobbs.farm"}}
 server -> {"m.server": "mess.hobbs.farm:443"}
 ```
 Under MAS, clients discover the auth server from Synapse's `auth_metadata` endpoint, not from `.well-known`, so **both files should stay as they are**. Verify a client can complete discovery end to end:
 ```bash
 curl -s https://hobbs.farm/.well-known/matrix/client
 curl -s https://mess.hobbs.farm/_matrix/client/unstable/org.matrix.msc2965/auth_metadata | head -c 200
 ```
 Expected: `.well-known` unchanged and still pointing at `mess.hobbs.farm`; auth_metadata returning the `auth.hobbs.farm` issuer. If a client cannot find the auth server, revisit `http.public_base` in the MAS config (Task 5 Step 2) before touching `.well-known`.
 - [ ] **Step 5: Add a Prometheus blackbox probe for the new endpoint**
 Add `https://auth.hobbs.farm/health` to the blackbox target list in `/mnt/docker-data/configs/prometheus/prometheus.yml`, then hot-reload:
 ```bash
 ssh glyptic@192.168.86.192 'curl -s -X POST http://localhost:9095/-/reload && sleep 5 && curl -s "http://localhost:9095/api/v1/query?query=probe_success" | grep -o "auth.hobbs.farm"'
 ```
 Expected: the target appears and probes `1`. **Auth being down now means nobody can log in — this probe is not optional.**
 - [ ] **Step 6: Commit**
 ```bash
 git add compose/ && git commit -m "feat: publish MAS at auth.hobbs.farm with blackbox probe"
 ```
 ---
 ### Task 9: Enable the QR MSCs
 **Files:**
 - Modify: `/var/lib/synapse/homeserver.yaml` (node only)
 - [ ] **Step 1: Enable both MSCs**
 ```yaml
 experimental_features:
  msc4108_enabled: true
  msc4388_mode: authenticated
 ```
 `msc4388_mode: authenticated` requires the rendezvous channel user to be logged in — prefer it over `open`, which would let unauthenticated parties open rendezvous sessions on an internet-facing server.
 - [ ] **Step 2: Restart and confirm Synapse actually starts**
 ```bash
 ssh glyptic@192.168.86.192 'docker restart synapse && sleep 45 && docker logs synapse --since 2m 2>&1 | grep -iE "ConfigError|MSC4108|MSC4388" | head'
 ```
 Expected: **no `ConfigError`**. If you see `MSC4108 requires matrix_authentication_service to be enabled`, Task 7 Step 3 did not take effect.
 - [ ] **Step 3: Verify the feature is advertised**
 ```bash
 ssh glyptic@192.168.86.192 'curl -s http://localhost:8008/_matrix/client/versions | python3 -c "
 import sys,json; uf=json.load(sys.stdin)[\"unstable_features\"]
 print(\"msc4108:\", uf.get(\"org.matrix.msc4108\"))
 "'
 ```
 Expected: `msc4108: True` (it is `False` today).
 - [ ] **Step 4: Commit**
 ```bash
 git commit --allow-empty -m "feat: enable MSC4108/MSC4388 QR sign-in"
 ```
 ---
 ### Task 10: End-to-end QR sign-in test and documentation
 - [ ] **Step 1: Test the actual feature**
 On an already-signed-in Element X Android device: Settings → Sessions → "Sign in with QR code" / "Link new device". On a second client, choose "Sign in with QR code" and scan.
 Expected: the new device signs in and receives E2EE keys **without** the password being typed. This is the deliverable — if it does not work here, the feature is not done regardless of what the config says.
 - [ ] **Step 2: Re-verify the full account inventory one more time**
 Re-run Task 7 Step 5. Expected: unchanged — all bots syncing, all tokens valid.
 - [ ] **Step 3: Complete the mandatory post-change checklist (CLAUDE.md)**
 - CLAUDE.md: MAS + two PostgreSQL containers on .192, port 8020, `auth.hobbs.farm`, Synapse now on PostgreSQL (supersedes Known Issues #24/#25 wording), new known issue for the MAS dependency — **auth is now a hard dependency for every login, including bots**
 - MEMORY.md: new entry for the MAS migration; update `matrix-account` (the reset recipe changes — user management moves to MAS)
 - Prometheus: done in Task 8 Step 4
 - Farm Manager links page: add `auth.hobbs.farm` to `serverLinks` in `/opt/farm-manager/server/static/app.js`, then **rebuild the image** (links are baked in, not mounted) and redeploy
 - Wiki: run the documentation-keeper agent
 - [ ] **Step 4: Retire the backups once stable**
 Leave `/var/lib/synapse.bak-premas/`, `/var/lib/synapse.bak-20260820-pre1159/`, and the SQLite file in place for **at least a week** of normal use. Only then delete.
 - [ ] **Step 5: Final commit**
 ```bash
 git add -A && git commit -m "docs: MAS QR sign-in deployment complete"
 ```
 ---
 ## Rollback Summary
 | Failed at | Recovery |
 |---|---|
 | Task 3 (port to PG) | Restore `homeserver.yaml.bak-presql`; SQLite file untouched and still authoritative |
 | Task 7 (syn2mas) | Restore `synapse-premas.sql.gz` + `mas-premigrate.sql.gz`; remove `matrix_authentication_service` block |
 | Task 9 (MSCs) | Remove `experimental_features` block; restart — no data implications |
 ## Open Questions for the Operator
 1. **Element Desktop compatibility.** Your two Element Desktop sessions are on the legacy auth stack. OIDC-native support in Element Web/Desktop has historically lagged Element X. Worth confirming before migrating, since a broken desktop client is a worse day than no QR code.
 2. **Is `auth.hobbs.farm` the hostname you want?** It becomes permanent — it is baked into session state, and changing `public_base` later invalidates sessions.
 3. **Vaultwarden is currently locked out** (memory `vault-master-password-stale`). MAS generates signing keys that must be backed up somewhere durable. Resolve the vault access first, or agree on an alternative.
 4. **Node memory.** .192 sits at ~14/23GB with Frigate as the main consumer. This adds ~1GB across three containers (Synapse's PG, MAS's PG, MAS). If that is too tight, the MAS stack could go on .100 instead — it only needs network reachability to Synapse, not co-location.
@@ -0,0 +1,70 @@
 # Matrix Auth Inventory — baseline before MAS migration
 Captured 2026-08-20 from `/var/lib/synapse/homeserver.db` (SQLite, pre-migration).
 **Totals:** 20 users (7 deactivated), **65 devices**, **54 access tokens**, 9 e2e device key entries.
 ## Per-account baseline
 | Account | Devices | Tokens | Password hash | Credential location | Auth style |
 |---|---|---|---|---|---|
 | `@jeremy` | 12 (incl. cross-signing keys) | 3 | yes | user-typed (`bigdog01`) | password + Element sessions |
 | `@clippyauto` | **17** | **17** | yes | n8n credential store (encrypted, `/home/node/.n8n/database.sqlite` on .100) | **repeated password login** — new device per login |
 | `@pluto` | **15** | **15** | yes | unknown (OpenClaw Gateway devices) | **repeated password login** |
 | `@julie` | 5 | 2 | yes | Element X Android | password |
 | `@spirit` | 2 | 3 | yes | `/mnt/infrastructure/spirit/openclaw.json` → key `accessToken` only | **access token** |
 | `@margaret` | 2 | 2 | yes | `/mnt/infrastructure/margaret/openclaw.json` → keys `accessToken` **and** `password` | token + password |
 | `@frigate` | 2 | 2 | yes | `compose/hf-pdocker-01/frigate-notify/.env` → `MATRIX_ACCESS_TOKEN=syt_ZnJpZ2F0…` | **access token** |
 | `@alertbot` | 2 | 2 | yes | `/mnt/docker-data/configs/alertmanager/matrix-receiver-config.yml` → `token: "syt_YWxlcnRib3Q…"` | **access token** |
 | `@hermes` | 2 | 2 | yes | unknown | unknown |
 | `@fry` | 1 | 1 | yes | unknown | unknown |
 | `@testbot` | 1 | 1 | yes | `TESTBOT_CI` device | unknown |
 | `@admin1`, `@adminuser` | 2 each | 2 each | yes | unknown | admin accounts |
 | `@aaron`, `@bob`, `@clippy`, `@travis`, `@tempadmin`, `@resetadmin`, `@resetadmin2` | 0 | 0 | none | — | deactivated |
 ## Risk read
 - **Access-token accounts** (`spirit`, `frigate`, `alertbot`) are the safest: syn2mas imports access tokens, so they should continue uninterrupted.
 - **`@clippyauto` and `@pluto` are the canaries.** The 1:1 device-to-token ratio and high counts mean they authenticate with a password on each connect, creating a fresh device every time. Post-migration they exercise the MAS **compatibility layer** (`/_matrix/client/*/login` proxied to MAS). If the bcrypt `unicode_normalization: true` scheme is misconfigured, these break first and loudest — `@clippyauto` polls every ~2s, so failure is visible within seconds.
 - `@jeremy`'s 12 devices include cross-signing keys (master/self_signing/user_signing), not just sessions. Cross-signing state must survive or Element will prompt to re-verify.
 ## Post-migration verification command
 ```bash
 ssh glyptic@192.168.86.192 'docker exec synapse-db psql -U synapse -d synapse -tc "
 select u.name,
        (select count(*) from devices d where d.user_id=u.name) dev,
        (select count(*) from access_tokens a where a.user_id=u.name) tok
 from users u where u.deactivated=0 order by u.name;"'
 ```
 Compare against the table above. Device counts may **grow** (new logins) but must not shrink for token-based accounts.
 ---
 ## syn2mas dry-run results (2026-08-20 20:07Z)
 `syn2mas check`: **"Check completed successfully with no errors or warnings."**
 Dry-run migration summary (rolled back automatically):
 ```
 20 users migrated (0 skipped)
 0 third-party IDs migrated (0 skipped)
 0 upstream links (external IDs) migrated (0 skipped)
 53 non-refreshable access tokens migrated (0 skipped)
 0 refreshable token pairs migrated (0 skipped)
 53 devices migrated (0 skipped)
 ```
 ### Reconciliation against the pre-migration baseline
 | Metric | Synapse | syn2mas | Explanation |
 |---|---|---|---|
 | Users | 20 | 20 | exact match |
 | Devices | 65 | 53 | 12 of the 65 are **cross-signing key pseudo-devices** (`display_name like '%signing key%'`), not login sessions. Real login devices = 53. Verified by query. Cross-signing keys stay in Synapse's own e2e tables. |
 | Access tokens | 54 | 53 | The one token not migrated is **device-less**, and MAS compat sessions are device-scoped. Identified as `syt_Ym9i_…` — `Ym9i` is base64 for **"bob"**, a leftover token from the decommissioned `@bob` agent, attached to `@spirit`. Last validated months ago. |
 **Spirit is safe:** its configured `accessToken` (`syt_c3Bpcml0…`, base64 `c3Bpcml0` = "spirit") maps to device `CEBHTAXUAY`, which migrates. Verified by joining the config token against `access_tokens`.
 **Conclusion: go.** No account loses a credential it actually uses.
Author	SHA1	Message	Date
jeremy	122ea392d1	infra: 2026-10-26 version audit, upgrades, UniFi re-removal - Synapse v1.163.2, MAS 1.27.1, Traefik v3.7, node-exporter v1.12.1 (all nodes), cadvisor -> ghcr.io v0.60.6, alertmanager-receiver 2026.9.23, Gitea 1.26.4, Jellyfin 12.3, Seerr v3.5.1 - Watchtower -> nickfedor/watchtower fork, twice daily (04:00/16:00) - Monitoring legacy node-exporter gated behind a compose profile - unifi-os-server on .100 removed again; compose file renamed so it cannot be revived - CLAUDE.md: Known Issues #40-43, service tree updates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NQhgEUqcaV4WM2S4hfXgHD	2026-10-26 16:27:43 -05:00
jeremy	f2374506b4	infra: pin hermes, add llamacpp backend + pinned agent Dockerfiles - hermes: pin nousresearch/hermes-agent:v2026.8.18 (was :latest, a rolling dev build that Watchtower never re-pulled -> 7 weeks stale). - llamacpp: add compose for the llama.cpp inference backend serving hermes on .150:8081 (q8_0 KV cache; 2.4x ollama on the same GGUF). - ai-agents: add reconstructed, version-pinned Dockerfiles for margaret and spirit (openclaw was built from @latest with no Dockerfile stored). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-10-06 19:06:06 -05:00
jeremy	beb727528a	docs: UniFi controller consolidation, STP fix, NFS recovery - Remove duplicate unifi-os-server container from .100; sole UniFi controller is now the .150 Podman instance (UI :11443, inform :8080). Data preserved; compose dir kept. - Document STP root-cause/fix for wifi client drops: version mismatch (stp vs rstp) + default priorities -> root flapping; fixed by RSTP everywhere + .159 pinned root @4096. - Known Issue #20: correct the stale .100:11443 blackbox probe note - the probe DID exist in the live config and was removed 2026-10-06 (backup saved) + Prometheus reloaded; UnifiDeviceDown rule is inert. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-10-06 19:06:05 -05:00
jeremy	9a81138cc9	docs: QR sign-in confirmed working end-to-end Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 16:36:27 -05:00
jeremy	84021278f9	docs: MAS deployment - CLAUDE.md service lists and known issues 24,27,28,29 - Synapse DB history rewritten (SQLite -> local -> PostgreSQL, and why) - #27: Synapse no longer serves /login; Traefik compat routing is load-bearing - #28: MAS tags are not v-prefixed; distroless; /health only on :8021 - #29: MAS signing keys need a vault home (vault currently locked out) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:21:21 -05:00
jeremy	a0bbf801f1	feat: enable MSC4108/MSC4388 - QR sign-in now advertised org.matrix.msc4108 = True (was False); rendezvous endpoint live (400 not 404). msc4388_mode: authenticated so unauthenticated internet parties cannot open rendezvous sessions. All bots syncing, 0 errors. New containers total ~263MB (mas 39MB, mas-db 40MB, synapse-db 184MB). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:18:10 -05:00
jeremy	8825cc3d4a	feat: publish MAS at auth.hobbs.farm; route login/logout/refresh to MAS Under MAS, Synapse returns M_UNRECOGNIZED for the login endpoints, so the mess-auth-compat router (priority 1000) sends /_matrix/client/*/login, /logout and /refresh to MAS while everything else stays on Synapse. Cloudflare A record for auth.hobbs.farm already existed (98.160.103.126, DNS-only). Blackbox probes added for MAS internal health (:8021) and the public discovery endpoint; all four matrix-related probes UP. Snapshot of the node's traefik dynamic config committed for reference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:15:57 -05:00
jeremy	9dff3c2d95	feat: synapse auth delegated to MAS; 20 users/53 tokens/53 devices migrated Verified: auth_metadata 200, migrated password authenticates via MAS compat layer, migrated access tokens still valid against Synapse, all bots syncing. Downtime 55s (20:09:17Z-20:10:12Z). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:12:27 -05:00
jeremy	f9a024ce10	docs: syn2mas dry-run results and count reconciliation Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:09:07 -05:00
jeremy	af2b560686	feat: deploy matrix authentication service 1.23.0 on :8020 Config at /mnt/docker-data/configs/mas/config.yaml (node-only, holds signing keys). Health on :8021 (distroless image - no container healthcheck possible). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:07:23 -05:00
jeremy	89eddfc134	chore: synapse ported sqlite -> postgres (20 users/93888 events/65 devices verified) Node-side config change; homeserver.yaml is not tracked in the repo. Downtime 2m35s. SQLite file retained inert at /var/lib/synapse/homeserver.db. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:04:32 -05:00
jeremy	d7ca70db32	feat: add mas-db postgres for matrix authentication service Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 15:01:45 -05:00
jeremy	a220e7903b	feat: add postgres 16 for synapse (local disk, C collation) ahead of MAS Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 14:59:35 -05:00
jeremy	986f219135	docs: matrix auth inventory and MAS QR sign-in plan Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-20 14:57:55 -05:00