├── [COMPOSE] clip-service (:5200) -- **RUNNING, and actively used.** CORRECTED 2026-09-21: the 2026-07-13 decommission note below was wrong for the current deployment — it was redeployed and the *live* `frigate-notify/docker-compose.yml` sets `CLIP_SERVICE_URL=http://localhost:5200`, with live `app/recognition.py` calling `/embed`. ~1.2GB RSS. **Do not remove without first rewiring frigate-notify's recognition pipeline.**
├── [COMPOSE] clip-service (:5200) -- **RUNNING, and actively used.** CORRECTED 2026-09-21: the 2026-07-13 decommission note below was wrong for the current deployment — it was redeployed and the *live* `frigate-notify/docker-compose.yml` sets `CLIP_SERVICE_URL=http://localhost:5200`, with live `app/recognition.py` calling `/embed`. ~1.2GB RSS. **Do not remove without first rewiring frigate-notify's recognition pipeline.**
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] ESPHome (:6052)
├── [COMPOSE] ESPHome (:6052) -- manages **Pool Value Controls** (ESP8266 at static **192.168.86.6**, drives pool pumps/valves; only ESPHome device registered in HA). Also polelightpond (.14) and living_room_lights (.153) configs, NOT in HA. See Known Issue #39.
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Synapse/Matrix (:8008) + wellknown (:8010) + synapse-db (**postgres:16-alpine, LOCAL /var/lib/synapse-db, C collation**) -- Synapse data on LOCAL /var/lib/synapse. Migrated SQLite->PostgreSQL 2026-08-20 (required by syn2mas). **Auth is delegated to MAS** -- Synapse no longer serves /login.
├── [COMPOSE] Synapse/Matrix (:8008) + wellknown (:8010) + synapse-db (**postgres:16-alpine, LOCAL /var/lib/synapse-db, C collation**) -- Synapse data on LOCAL /var/lib/synapse. Migrated SQLite->PostgreSQL 2026-08-20 (required by syn2mas). **Auth is delegated to MAS** -- Synapse no longer serves /login.
├── [COMPOSE] **MAS / Matrix Authentication Service (:8020 web, :8021 health) + mas-db (postgres:16-alpine, LOCAL /var/lib/mas-db)** -- OAuth2/OIDC provider for Matrix. Public: auth.hobbs.farm. Deployed 2026-08-20 to enable QR sign-in (MSC4108). QR sign-in **verified working end-to-end 2026-08-20**. **Every Matrix login now depends on this service.**
├── [COMPOSE] **MAS / Matrix Authentication Service (:8020 web, :8021 health) + mas-db (postgres:16-alpine, LOCAL /var/lib/mas-db)** -- OAuth2/OIDC provider for Matrix. Public: auth.hobbs.farm. Deployed 2026-08-20 to enable QR sign-in (MSC4108). QR sign-in **verified working end-to-end 2026-08-20**. **Every Matrix login now depends on this service.**
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090) -- `traefik:v3.7` (3.7.16, bumped from v3.6 2026-10-26; 3.7 migration notes are k8s-only for us)
├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192.
├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192.
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
├── [COMPOSE] **Onyx (RAG/search) -- 10 containers, `/mnt/docker-data/compose/hf-pdocker-01/onyx`.** NOT decommissioned (the bart entry below refers to an older, separate deployment). Public: **onyx.hobbs.farm** -> nginx :3005. Custom build: `gitea_patch.py` + `unstructured_patch.py` index Gitea (`git.hobbs.farm/api/v1`) and call unstructured-api on .100:8000. LLM -> `192.168.86.150:18020/v1` (`qwen3.8-27b`) — **dead while .150 is down.** `onyx-opensearch-1` **stopped 2026-09-21** (`ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false`, held no Onyx data, nothing `depends_on` it; freed ~1.1GB). Restart with `docker compose up -d opensearch` if the flag is ever flipped.
├── [COMPOSE] **Onyx (RAG/search) -- 10 containers, `/mnt/docker-data/compose/hf-pdocker-01/onyx`.** NOT decommissioned (the bart entry below refers to an older, separate deployment). Public: **onyx.hobbs.farm** -> nginx :3005. Custom build: `gitea_patch.py` + `unstructured_patch.py` index Gitea (`git.hobbs.farm/api/v1`) and call unstructured-api on .100:8000. LLM -> `192.168.86.150:18020/v1` (`qwen3.8-27b`) — **dead while .150 is down.** `onyx-opensearch-1` **stopped 2026-09-21** (`ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false`, held no Onyx data, nothing `depends_on` it; freed ~1.1GB). **2026-10-10: it came back on its own after .192's crash/reboot (1.5GB) — now stopped, `restart=no`, and gated behind compose `profiles: ["opensearch"]` so neither reboots nor `compose up` revive it.** Restart with `docker compose --profile opensearch up -d opensearch` if the flag is ever flipped.
├── [COMPOSE] hermes -- **moved here from .150** (host network, `/mnt/docker-data/compose/hf-pdocker-01/hermes`, pinned `v2026.8.18`). Points at `http://192.168.86.150:18020/v1` model `qwen3.8-27b`; that endpoint is **dead**, so every call burns 200s x3 retries and fails. Note this is a different host:port than the `.150:8081` llamacpp setup documented in the GPU Machine section.
├── [COMPOSE] hermes -- **moved here from .150** (host network, `/mnt/docker-data/compose/hf-pdocker-01/hermes`, pinned `v2026.8.18`). Points at `http://192.168.86.150:18020/v1` model `qwen3.8-27b`; that endpoint is **dead**, so every call burns 200s x3 retries and fails. Note this is a different host:port than the `.150:8081` llamacpp setup documented in the GPU Machine section.
├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
├── [COMPOSE] synapse-admin -- Synapse admin UI
├── [COMPOSE] synapse-admin -- Synapse admin UI
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)
bart (15GB RAM, i5-4570T, HDD)
bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12.
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, **Seerr** -- replaced Jellyseerr 2026-10-17, `seerr/seerr:v3.5.1`, config LOCAL /var/lib/seerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin on official STABLE 12.x, pinned to the dated tag `jellyfin/jellyfin:12.3.20261023-013714` (12.3.0, 2026-10-26).** History: ran the 2026060106 unstable build from 2026-07-14 because the DB had been migrated to the v12 schema (10.11.x cannot read it); stable 12.0 shipped 2026-09-08, moved to 12.2 on 2026-10-17, 12.3 on 2026-10-26 (DB backup `/var/lib/jellyfin-backup-20261026-pre123`). **Never go below 12.**
├── [REMOVED] Onyx -- this *bart* deployment was decommissioned 2026-07-13. **A different, newer Onyx now runs on hf-pdocker-01 (see above) — 'Onyx is gone' is false.**
├── [REMOVED] Onyx -- this *bart* deployment was decommissioned 2026-07-13. **A different, newer Onyx now runs on hf-pdocker-01 (see above) — 'Onyx is gone' is false.**
├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15.
├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15.
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26)
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [REMOVED] UniFi OS Server -- the `unifi-os-server` container (ghcr.io/lemker/unifi-os-server, v5.1.15) was decommissioned 2026-10-05. It was a **duplicate/idle** controller -- the real UniFi controller is the **Podman UniFi OS Server on the Windows box (.150)**, which manages all UniFi devices (e.g. `downstairsswitch` US-24-250W at 192.168.86.159, inform `http://192.168.86.150:8080/inform`). Container + image + network removed; ports 11443/8080/3478/etc freed on .100. **Data preserved** at `/var/lib/unifi-os/*` (971M local) + `/mnt/docker-data/configs/unifi-os/persistent` (NAS); compose dir kept for reference. **Only .150 manages UniFi, by design.**
├── [REMOVED] UniFi OS Server -- the `unifi-os-server` container (ghcr.io/lemker/unifi-os-server, v5.1.15) was decommissioned 2026-10-05. It was a **duplicate/idle** controller -- the real UniFi controller is the **Podman UniFi OS Server on the Windows box (.150)**, which manages all UniFi devices (e.g. `downstairsswitch` US-24-250W at 192.168.86.159, inform `http://192.168.86.150:8080/inform`). Container + image + network removed; ports 11443/8080/3478/etc freed on .100. **Data preserved** at `/var/lib/unifi-os/*` (971M local) + `/mnt/docker-data/configs/unifi-os/persistent` (NAS); compose dir kept for reference. **Only .150 manages UniFi, by design.** **2026-10-26: it was found RUNNING again** (the 10-05 removal never stuck -- journal shows it up again later that same day, then recreated from its compose dir 2026-10-17 21:06Z). Removed again (container, `unifi_default` network, `ghcr.io/lemker/unifi-os-server` + stale `linuxserver/unifi-network-application` images) and the compose file renamed to `docker-compose.yml.decommissioned-20261026` + `DECOMMISSIONED.txt` so a stray `compose up` cannot revive it. Data still preserved.
├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
├── [COMPOSE] Vaultwarden, MCPO, Trivy
├── [COMPOSE] Vaultwarden, MCPO, Trivy
├── [COMPOSE] SearXNG (`searxng/searxng:latest`) -- found running 2026-10-26, previously undocumented here
├── [COMPOSE] unstructured-api (:8000, `downloads.unstructured.io/unstructured-io/unstructured-api:latest`) -- used by Onyx's `unstructured_patch.py` on .192
├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget)
├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget)
4.**Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
4.**Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
5.**Farm Manager links page** (last audited 2026-08-20: dropped dead Clippy/Bob/Aaron links, added FRIDAY Voice + LinkedStorm + Rongorongo and a new `GPU Machine (192.168.86.150)` category; `hermes` intentionally absent — it publishes no ports. Served path is `/app.js`, not `/static/app.js`.) — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)
5.**Farm Manager links page** (last audited **2026-10-26**: every link probed; removed both FRIDAY Voice links, added Onyx (public + :3005), Guacamole :8199, Synapse Admin :8080, SearXNG :8888, n8n :5678, CouchDB Fauxton; MCPO -> `/docs`, Trivy -> `/healthz` (API only, no UI); Jellyseerr -> Seerr. Groups live in `/mnt/docker-data/configs/farm-manager/groups.json` -- **update them too when containers are added/renamed**. Previous audit 2026-08-20: dropped dead Clippy/Bob/Aaron links, added FRIDAY Voice + LinkedStorm + Rongorongo and a new `GPU Machine (192.168.86.150)` category; `hermes` intentionally absent — it publishes no ports. Served path is `/app.js`, not `/static/app.js`.) — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)
Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.
Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.
@@ -305,7 +307,7 @@ All persistent data on NAS at `/mnt/docker-data/`:
2. **Telegraf Docker Input**: Disabled due to Docker socket permission issues across nodes
2. **Telegraf Docker Input**: Disabled due to Docker socket permission issues across nodes
3. **Watchtower API Version**: Requires `DOCKER_API_VERSION=1.44`
3. **Watchtower API Version**: Requires `DOCKER_API_VERSION=1.44`
4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running
4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running
5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
5. **Frigate cameras**: 192.168.44.0/24 is the **"Camera" VLAN 200** on the Firewalla (gateway 192.168.86.1 / 44.1). Cameras with 44.x IPs MUST sit on Camera-native switch ports. **2026-10-08:** house_north/.51, house_east/.58, house_south/.53, house_west/.52 are wired to downstairsswitch (.159) ports 3/4/18/19 (verified from the switch MAC table 2026-10-11), which are **VLAN 1**, so they can't be reached by routing. They only half-worked because .192's Wi-Fi NIC held a wrong-VLAN 44.x lease. The fix is to make those ports Camera-native (see memory `unifi-estate`). The nodes' `wlp2s0` Wi-Fi NICs must hold **86.x** leases; a 44.x lease there is stale and breaks routing to the real Camera VLAN. **2026-10-10: with the Wi-Fi workaround gone (86.x lease since 10-08), those 4 cameras are now fully DOWN (0 fps, no ping) until the ports are re-profiled.**
6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
7. **systemd-resolved DNSStubListener**: disabled on hf-pdocker-01 and hf-pdocker-02 via `/etc/systemd/resolved.conf.d/nostub.conf` (`DNSStubListener=no`) — originally for AdGuard Home's port 53. AdGuard was decommissioned 2026-07-13; the stub-listener change is harmless and left in place (re-enable only if something needs 127.0.0.53).
7. **systemd-resolved DNSStubListener**: disabled on hf-pdocker-01 and hf-pdocker-02 via `/etc/systemd/resolved.conf.d/nostub.conf` (`DNSStubListener=no`) — originally for AdGuard Home's port 53. AdGuard was decommissioned 2026-07-13; the stub-listener change is harmless and left in place (re-enable only if something needs 127.0.0.53).
8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`.
8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`.
@@ -327,7 +329,7 @@ All persistent data on NAS at `/mnt/docker-data/`:
23. **Matrix registration was wide open to the internet (fixed 2026-08-20)**: `mess.hobbs.farm` is publicly reachable (Traefik, port 443, federation + client API on :8008) and `homeserver.yaml` had `enable_registration: true` + `enable_registration_without_verification: true` — the registration flow was a bare `m.login.dummy`, i.e. **anyone on the internet could create an account on hobbs.farm with no email, captcha, or token**. No abuse had occurred (all 18 accounts are known/expected; newest legit account was @fry 2026-07-05). Fixed by setting both to `false` and `restrict_public_rooms_to_local_users: true` (was `false`, which published the public room directory to any remote server). Backup: `/var/lib/synapse/homeserver.yaml.bak-20260820-registration`. New accounts are still creatable on the box via `docker exec synapse register_new_matrix_user -k <registration_shared_secret>`.
23. **Matrix registration was wide open to the internet (fixed 2026-08-20)**: `mess.hobbs.farm` is publicly reachable (Traefik, port 443, federation + client API on :8008) and `homeserver.yaml` had `enable_registration: true` + `enable_registration_without_verification: true` — the registration flow was a bare `m.login.dummy`, i.e. **anyone on the internet could create an account on hobbs.farm with no email, captcha, or token**. No abuse had occurred (all 18 accounts are known/expected; newest legit account was @fry 2026-07-05). Fixed by setting both to `false` and `restrict_public_rooms_to_local_users: true` (was `false`, which published the public room directory to any remote server). Backup: `/var/lib/synapse/homeserver.yaml.bak-20260820-registration`. New accounts are still creatable on the box via `docker exec synapse register_new_matrix_user -k <registration_shared_secret>`.
24. **Synapse database history (SQLite -> local disk -> PostgreSQL, all 2026-08-20)**: Synapse ran on **SQLite on NFS** while CLAUDE.md claimed it used a `synapse-db` postgres that in fact had **0 tables**. First the data was moved to LOCAL `/var/lib/synapse` and the empty postgres deleted; then, because `syn2mas` **only reads PostgreSQL** ("The migration tool only supports reading from PostgreSQL for the Synapse database as well"), a real `synapse-db` was created and the DB ported with `synapse_port_db` (20 users / 93,888 events / 65 devices, verified identical; 2m35s downtime). The PG database MUST have `LC_COLLATE=C` -- verify with `select datcollate from pg_database where datname='synapse'` (PG16 removed the `SHOW lc_collate` GUC). The old SQLite file is still at `/var/lib/synapse/homeserver.db`, now inert; NFS copy at `/mnt/docker-data/configs/matrix/data` is an older backup -- **never remount either**.
24. **Synapse database history (SQLite -> local disk -> PostgreSQL, all 2026-08-20)**: Synapse ran on **SQLite on NFS** while CLAUDE.md claimed it used a `synapse-db` postgres that in fact had **0 tables**. First the data was moved to LOCAL `/var/lib/synapse` and the empty postgres deleted; then, because `syn2mas` **only reads PostgreSQL** ("The migration tool only supports reading from PostgreSQL for the Synapse database as well"), a real `synapse-db` was created and the DB ported with `synapse_port_db` (20 users / 93,888 events / 65 devices, verified identical; 2m35s downtime). The PG database MUST have `LC_COLLATE=C` -- verify with `select datcollate from pg_database where datname='synapse'` (PG16 removed the `SHOW lc_collate` GUC). The old SQLite file is still at `/var/lib/synapse/homeserver.db`, now inert; NFS copy at `/mnt/docker-data/configs/matrix/data` is an older backup -- **never remount either**.
25. **Synapse upgraded 1.148.0 -> 1.159.0 and moved to the element-hq registry (2026-08-20)**: it had been on `matrixdotorg/synapse:latest`, an **abandoned Docker Hub repo** whose `:latest` froze at 1.148.0 (image built 2026-02-24) — a moving tag that silently stopped moving on a publicly-exposed homeserver. Now pinned to `ghcr.io/element-hq/synapse:v1.159.0` (note the **`v` prefix** — `:1.159.0` without it 404s). Upgrade was clean: schema 93 -> 94, all background updates finished in ~5s, 0 errors, 19 users / 93,884 events intact, memory 112MB/1G. None of the v1.149-v1.159 breaking changes applied to us (MSC3861 removal — unused; `quarantined_media_changes` stream writer — workers only; the rest are Debian/Ubuntu packaging). Pre-upgrade backup: `/var/lib/synapse.bak-20260820-pre1159` (646MB) — delete once you're satisfied. **Pinned on purpose: bump it manually and deliberately, since the last moving tag rotted for 6 months unnoticed.**
25. **Synapse upgraded 1.148.0 -> 1.159.0 and moved to the element-hq registry (2026-08-20)**: it had been on `matrixdotorg/synapse:latest`, an **abandoned Docker Hub repo** whose `:latest` froze at 1.148.0 (image built 2026-02-24) — a moving tag that silently stopped moving on a publicly-exposed homeserver. Now pinned to `ghcr.io/element-hq/synapse:v1.159.0` (note the **`v` prefix** — `:1.159.0` without it 404s). Upgrade was clean: schema 93 -> 94, all background updates finished in ~5s, 0 errors, 19 users / 93,884 events intact, memory 112MB/1G. None of the v1.149-v1.159 breaking changes applied to us (MSC3861 removal — unused; `quarantined_media_changes` stream writer — workers only; the rest are Debian/Ubuntu packaging). Pre-upgrade backup: `/var/lib/synapse.bak-20260820-pre1159` (646MB) — delete once you're satisfied. **Pinned on purpose: bump it manually and deliberately, since the last moving tag rotted for 6 months unnoticed.** **2026-10-26: bumped v1.159.0 -> v1.163.2** (schema 94 deltas applied in <1s; config contains none of the options affected by the 1.160-1.163 upgrade notes). pg_dump backups taken first: `/var/lib/upgrade-backups-20261026/{synapse-db,mas-db}.dump` on .192. MAS bumped 1.23.0 -> **1.27.1** at the same time.
26. **`SYNAPSE_SERVER_NAME` in the matrix compose disagrees with homeserver.yaml**: compose passes `SYNAPSE_SERVER_NAME=mess.hobbs.farm` but the real `server_name` is **`hobbs.farm`** (user IDs are `@user:hobbs.farm`). The env var is only read when the image *generates* a config, so it's inert today — but if `homeserver.yaml` ever went missing, the container would generate a fresh config with the wrong `server_name` AND a new signing key, which would break federation and every existing user ID. Left as-is (changing it has no effect while the config exists); just never let the container auto-generate..
26. **`SYNAPSE_SERVER_NAME` in the matrix compose disagrees with homeserver.yaml**: compose passes `SYNAPSE_SERVER_NAME=mess.hobbs.farm` but the real `server_name` is **`hobbs.farm`** (user IDs are `@user:hobbs.farm`). The env var is only read when the image *generates* a config, so it's inert today — but if `homeserver.yaml` ever went missing, the container would generate a fresh config with the wrong `server_name` AND a new signing key, which would break federation and every existing user ID. Left as-is (changing it has no effect while the config exists); just never let the container auto-generate..
27. **Matrix auth is delegated to MAS -- Synapse does NOT serve /login any more (2026-08-20)**: with `matrix_authentication_service.enabled: true`, Synapse answers `M_UNRECOGNIZED` on `/_matrix/client/*/login`. The **reverse proxy must route the three compat paths to MAS or nobody can sign in**: `login`, `logout`, `refresh` (verified: MAS returns 200/405 on those and 404 on `register`/`whoami`, which stay on Synapse). Implemented as the `mess-auth-compat` router in `/mnt/docker-data/traefik/dynamic/temp-routes.yml` with explicit `priority: 1000` so it beats the catch-all `mess` router. **Anything that talks to `192.168.86.192:8008` directly bypasses Traefik and therefore cannot log in with a password** -- token-based bots (spirit, frigate, alertbot) are unaffected because they never call /login, but `@clippyauto` (n8n) and `@pluto` historically re-login with a password (17 and 15 accumulated devices). They currently run on migrated tokens; if either ever needs a fresh password login, point it at `https://mess.hobbs.farm` instead of `:8008`.
27. **Matrix auth is delegated to MAS -- Synapse does NOT serve /login any more (2026-08-20)**: with `matrix_authentication_service.enabled: true`, Synapse answers `M_UNRECOGNIZED` on `/_matrix/client/*/login`. The **reverse proxy must route the three compat paths to MAS or nobody can sign in**: `login`, `logout`, `refresh` (verified: MAS returns 200/405 on those and 404 on `register`/`whoami`, which stay on Synapse). Implemented as the `mess-auth-compat` router in `/mnt/docker-data/traefik/dynamic/temp-routes.yml` with explicit `priority: 1000` so it beats the catch-all `mess` router. **Anything that talks to `192.168.86.192:8008` directly bypasses Traefik and therefore cannot log in with a password** -- token-based bots (spirit, frigate, alertbot) are unaffected because they never call /login, but `@clippyauto` (n8n) and `@pluto` historically re-login with a password (17 and 15 accumulated devices). They currently run on migrated tokens; if either ever needs a fresh password login, point it at `https://mess.hobbs.farm` instead of `:8008`.
@@ -343,3 +345,11 @@ All persistent data on NAS at `/mnt/docker-data/`:
34. **Onyx self-registration was open to the internet (fixed 2026-09-21)**: `onyx.hobbs.farm` is publicly reachable and Onyx runs `AUTH_TYPE=basic` with **no** `REQUIRE_EMAIL_VERIFICATION` and **no** `VALID_EMAIL_DOMAINS` — so `POST /api/auth/register` let anyone on the internet create an account. External scanners were already probing the host (a DigitalOcean IP hit `/` on 2026-09-21). No abuse had occurred: only 2 accounts existed (`glyptic@gmail.com` + Onyx's internal `anonymous@onyx.app`). **Fix: a Traefik router `onyx-register-lockdown` (`priority: 1000`, beats the catch-all `onyx` router) matching `PathPrefix(/api/auth/register)` behind an `ipAllowList` middleware limited to `192.168.86.0/24` + `127.0.0.1/32`.** Verified by temporarily narrowing the range: register returned **403** from a non-allowlisted source while `/api/auth/login` (405) and `/` (200) were unaffected; range then restored. Config `/mnt/docker-data/traefik/dynamic/onyx.yml` (backup `.bak-20260920-195851`). **To add a user: register from the LAN, or widen the `sourceRange`.** Note `VALID_EMAIL_DOMAINS` was rejected as the control because it is domain-level only — `gmail.com` would admit any Gmail user. Same class as Known Issue #23 (open Matrix registration).
34. **Onyx self-registration was open to the internet (fixed 2026-09-21)**: `onyx.hobbs.farm` is publicly reachable and Onyx runs `AUTH_TYPE=basic` with **no** `REQUIRE_EMAIL_VERIFICATION` and **no** `VALID_EMAIL_DOMAINS` — so `POST /api/auth/register` let anyone on the internet create an account. External scanners were already probing the host (a DigitalOcean IP hit `/` on 2026-09-21). No abuse had occurred: only 2 accounts existed (`glyptic@gmail.com` + Onyx's internal `anonymous@onyx.app`). **Fix: a Traefik router `onyx-register-lockdown` (`priority: 1000`, beats the catch-all `onyx` router) matching `PathPrefix(/api/auth/register)` behind an `ipAllowList` middleware limited to `192.168.86.0/24` + `127.0.0.1/32`.** Verified by temporarily narrowing the range: register returned **403** from a non-allowlisted source while `/api/auth/login` (405) and `/` (200) were unaffected; range then restored. Config `/mnt/docker-data/traefik/dynamic/onyx.yml` (backup `.bak-20260920-195851`). **To add a user: register from the LAN, or widen the `sourceRange`.** Note `VALID_EMAIL_DOMAINS` was rejected as the control because it is domain-level only — `gmail.com` would admit any Gmail user. Same class as Known Issue #23 (open Matrix registration).
35. **The live Prometheus config is `/mnt/docker-data/configs/prometheus/prometheus.yml` — the copy at `compose/hf-pdocker-01/monitoring/prometheus.yml` is STALE and misleading (2026-09-21)**: the compose-dir copy still lists decommissioned targets, including Onyx at the wrong node (`192.168.86.167:3005`, bart) and OpenVAS (`:9392`). Neither is in the live config, so neither was ever probed — grepping the compose copy to answer "is X monitored?" gives the wrong answer. Onyx had **no probe at all** until one was added to the live file on 2026-09-21 (`http://192.168.86.192:3005`, now `probe_success=1`). Backup `prometheus.yml.bak-20260920-*`. The live file is also the better changelog: it records the voicecoach (2026-08-24) and FRIDAY Voice / speaches (2026-08-26) teardowns that CLAUDE.md still described as live.
35. **The live Prometheus config is `/mnt/docker-data/configs/prometheus/prometheus.yml` — the copy at `compose/hf-pdocker-01/monitoring/prometheus.yml` is STALE and misleading (2026-09-21)**: the compose-dir copy still lists decommissioned targets, including Onyx at the wrong node (`192.168.86.167:3005`, bart) and OpenVAS (`:9392`). Neither is in the live config, so neither was ever probed — grepping the compose copy to answer "is X monitored?" gives the wrong answer. Onyx had **no probe at all** until one was added to the live file on 2026-09-21 (`http://192.168.86.192:3005`, now `probe_success=1`). Backup `prometheus.yml.bak-20260920-*`. The live file is also the better changelog: it records the voicecoach (2026-08-24) and FRIDAY Voice / speaches (2026-08-26) teardowns that CLAUDE.md still described as live.
36. **SSH key auth to hf-pdocker-01 is broken — password only (2026-09-21)**: all of `~/.ssh/id_docker_server`, `id_bart`, `id_nas` are rejected by `192.168.86.192` (`Permission denied (publickey,password)`), while `id_docker_server` still works for `.100` and `id_bart` for bart. Connect with `sshpass -p '&Rip8a115' ssh -o PubkeyAuthentication=no glyptic@192.168.86.192`. **Any key-based automation targeting .192 is silently failing** — check `~/.ssh/authorized_keys` on that node (likely lost when the host was rebuilt/restored).
36. **SSH key auth to hf-pdocker-01 is broken — password only (2026-09-21)**: all of `~/.ssh/id_docker_server`, `id_bart`, `id_nas` are rejected by `192.168.86.192` (`Permission denied (publickey,password)`), while `id_docker_server` still works for `.100` and `id_bart` for bart. Connect with `sshpass -p '&Rip8a115' ssh -o PubkeyAuthentication=no glyptic@192.168.86.192`. **Any key-based automation targeting .192 is silently failing** — check `~/.ssh/authorized_keys` on that node (likely lost when the host was rebuilt/restored).
37. **hf-pdocker-01 hard-hung for ~27h (2026-10-09 13:55 → 2026-10-10 17:08 CDT)**: journal simply stops at 13:55 with no OOM, kernel error, or shutdown record; `sar` at 13:50 showed a healthy box (62% mem, 23% swap, load ~4), and .100/bart stayed up (9d uptime), so it was neither memory exhaustion nor a house-wide power event. Came back on kernel 7.0.0-38 (was -34) after a power cycle. Cause unknown — if it recurs, suspect hardware (this box was repaired 2026-03) and consider enabling persistent `kdump`/`netconsole`. Side effects found on 2026-10-10: `onyx-opensearch-1` revived itself (fixed, see Onyx entry). Also fixed then: `ServiceHighMemory` used `container_memory_usage_bytes`, which counts reclaimable page cache, so clip-service false-alarmed at 94% while really at 64% — rule now uses `container_memory_working_set_bytes` (backup `alerts.yml.bak-20261010-workingset`).
38. **bart shows ~20 nameless `Dead` containers in `docker ps -a`** — orphaned `/var/lib/docker/containers/<id>` dirs for containers whose images Watchtower already deleted. `docker rm` says "No such container". Harmless clutter; clearing them requires stopping dockerd (moving the dirs aside) which restarts the whole media stack, so it was left for a maintenance window.
39. **[RESOLVED 2026-10-11] Static-IP ESPHome boards were invisible behind UniFi Proxy ARP (found 2026-10-10)**: the `ASUS_F0_2G` WLAN has **Proxy ARP on** (`proxy_arp=1` in every AP's `/etc/hostapd/wifi0ap0.cfg`; it also has `ap_isolate=1` and `mcast.enhance=1`). With Proxy ARP, the AP answers ARP for wireless clients **from a table it learns by snooping DHCP** and stops passing ARP broadcasts through to them. Boards with ESPHome `manual_ip` never DHCP, so after any reconnect nobody answers ARP for them: they associate fine and even announce themselves over mDNS, but are unreachable. **Pool Value Controls (.6, MAC `a4:cf:12:f1:6e:c8`) went dark 2026-10-09 11:04** at a Wi-Fi reconnect on AP .187 and then looped every 15 min on ESPHome's API `reboot_timeout`. Diagnosed by capture on .187: ARP who-has .6 reached the radio and got no reply, while a static neighbor entry made ping and API work at once. **Workaround (live):** systemd unit `static-arp-esphome.service` on .192 pins `.6 -> a4:cf:12:f1:6e:c8 nud permanent` on `enp0s31f6` (HA and ESPHome are bridge-networked on .192, so they use the host's neighbor table). **Real fix (pending, needs controller access on .150):** turn Proxy ARP off on the `ASUS_F0_2G` WLAN, OR switch the boards to DHCP + Firewalla reservations. Once Proxy ARP is off, `systemctl disable --now static-arp-esphome` on .192. A `power_save_mode: none` change was also OTA'd to the board (backup `pool-value-controls.yaml.bak-20261010-powersave`). It did NOT fix this, but it's harmless and was left in. **Any other static-IP Wi-Fi device on that SSID has the same problem.** **RESOLVED 2026-10-11:** Proxy ARP (and `ap_isolate`) were turned off on `ASUS_F0_2G` in the controller (pushed to all APs 11:09–11:18; the `proxy_arp`/`ap_isolate` lines are gone from every AP's hostapd config). `static-arp-esphome.service` was then disabled and deleted on .192. Verified afterwards: .6 resolves by normal ARP (neighbor `REACHABLE`, not `PERMANENT`), ping OK, HA API session established. If a static-IP Wi-Fi device goes dark again, check that Proxy ARP hasn't been switched back on. polelightpond (.14) and living_room_lights (.153) were still unreachable after the fix, so they have some other problem (unpowered or offline).
40. **Container version audit + upgrade round (2026-10-26)**. Upgraded: Synapse v1.163.2, MAS 1.27.1, Traefik v3.7 (3.7.16), node-exporter v1.12.1 (all 3 nodes), cadvisor v0.60.6, matrix-alertmanager-receiver 2026.9.23, Gitea 1.26.4 (patch only; pg_dump at `~/gitea-db-20261026-pre1264.dump` on .100), Jellyfin 12.3, Seerr v3.5.1, plus re-pulls of every moving tag on .192 (HA 2026.10.4, ESPHome, Mosquitto, Prometheus, Blackbox, Telegraf, nginx, litellm, and the postgres/mysql/redis patch images). Every compose file touched has a `.bak-20261026` next to it on the NAS. **Deliberately NOT done, each needs its own decision:** Onyx `:latest` now = **v5.0.0 (major, released 2026-10-24)**, still on v4.9.2 (no Watchtower on .192, so it only moves on a manual pull); Gitea 1.27.x / **28.x** (major jumps); OpenClaw 2026.7.1-2 -> 2026.10.7 (one-way state migration, see #30); hermes v2026.8.18 -> v0.22.5 (upstream switched to semver tags, and its LLM endpoint is dead anyway); **MySQL 8.0 (guacamole-db, semaphore-db) went EOL April 2026 -> needs an 8.4 LTS migration with a dump/restore**; InfluxDB 1.8 (EOL); Onyx's `postgres:15.2-alpine` (2023 patch level, managed by Onyx's compose); `awesometechnologies/synapse-admin` (stale, maintained fork is `etkecc/synapse-admin`). How to audit: compare each container's `RepoDigests` against registry HEAD digests -- for Docker Hub use the hub.docker.com web API (`/v2/namespaces/<ns>/repositories/<repo>/tags/<tag>` -> `digest`), which does not count against pull limits. `docker buildx imagetools inspect --format` is ignored by the nodes' buildx v0.31.1.
41. **Docker Hub anonymous pull limit is shared by the whole house** (public IP 98.160.103.126, 100 pulls/hour): on 2026-10-26 every node AND WSL got 429 `toomanyrequests`, meaning Watchtower updates of Docker Hub images were silently failing. Mitigation applied: both Watchtowers moved from a 24h poll interval to a **twice-daily cron schedule** (04:00/16:00), and the archived `containrrr/watchtower` swapped for the maintained drop-in fork `nickfedor/watchtower` (**image name is `nickfedor/`, not `nicholas-fedor/`** -- the latter is the GitHub org and 404s on Hub). Same env vars and still honors `com.centurylinklabs.watchtower.enable`. A `docker login` on the nodes would raise the limit further. **cadvisor no longer publishes to gcr.io** after v0.55 -- the image is now `ghcr.io/google/cadvisor`.
42. **node-exporter had been launched from `/tmp` (compose project `tmp`) on all 3 nodes**, so `docker compose` in the real dirs didn't own it. Fixed 2026-10-26: removed and redeployed from `/mnt/docker-data/compose/<node>/node-exporter/`. Separately, the **monitoring** compose on .192 still defines a legacy `node-exporter:latest` service on :9100 that a bare `docker compose up -d` would try to start; it is now gated behind `profiles: ["legacy-node-exporter"]`. Also: the matrix-alertmanager-receiver container is now named `monitoring-matrix-alertmanager-receiver-1` (compose default) -- Alertmanager reaches it by service name `matrix-alertmanager-receiver`, which still resolves.
43. **Farm Manager showed ZERO containers for hf-pdocker-01 (fixed 2026-10-26)**: the server fetches each agent's `/containers` with a timeout and on any error silently returns `[]` for that node. .192's agent took ~12s (47 containers, per-container `stats(stream=False)` = Docker samples twice ~1s apart, only 10 workers) vs a 10s timeout, so the main host's containers simply vanished from the dashboard while `/api/nodes` still said "healthy, 47". Fix: agent uses `stats(..., one_shot=True)` + 20 workers (now ~2s), server timeout 30s. Agent rebuilt on .192 and shipped to .100/bart via `docker save | docker load` (only .192 has the `/opt/farm-manager` source); rollback tags `farm-agent:rollback-20261026` / `farm-manager:rollback-20261026`. Same day: removed a dead `gpu-workstation` node (192.168.86.27) from the live `config.json`, made `/health` ignore Dead containers (bart reported 30 instead of 10), and rebuilt `groups.json` -- every group still used Swarm-era / decommissioned names (`ai-agents_margaret`, `jellyseerr`, `adguardhome`, bart Onyx, `openvas`, `unifi`...), so all group actions were no-ops. Now 14 groups / 63 containers; per-node utilities (farm-agent, node-exporter, telegraf, watchtower) are deliberately ungrouped so a group "stop" can't kill the agents. **4 tests in `tests/test_agent_docker_ops.py` fail -- pre-existing** (they mock `containers.list` but the code uses low-level `api.containers`); not caused by this change.
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
