diff --git a/CLAUDE.md b/CLAUDE.md index 7953cfc..b8f07e5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,13 +25,13 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha ├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications ├── [COMPOSE] clip-service (:5200) -- **RUNNING, and actively used.** CORRECTED 2026-09-21: the 2026-07-13 decommission note below was wrong for the current deployment — it was redeployed and the *live* `frigate-notify/docker-compose.yml` sets `CLIP_SERVICE_URL=http://localhost:5200`, with live `app/recognition.py` calling `/embed`. ~1.2GB RSS. **Do not remove without first rewiring frigate-notify's recognition pipeline.** ├── [COMPOSE] Home Assistant (:8123) -├── [COMPOSE] ESPHome (:6052) +├── [COMPOSE] ESPHome (:6052) -- manages **Pool Value Controls** (ESP8266 at static **192.168.86.6**, drives pool pumps/valves; only ESPHome device registered in HA). Also polelightpond (.14) and living_room_lights (.153) configs, NOT in HA. See Known Issue #39. ├── [COMPOSE] Mosquitto (:1883, :8883) ├── [COMPOSE] Synapse/Matrix (:8008) + wellknown (:8010) + synapse-db (**postgres:16-alpine, LOCAL /var/lib/synapse-db, C collation**) -- Synapse data on LOCAL /var/lib/synapse. Migrated SQLite->PostgreSQL 2026-08-20 (required by syn2mas). **Auth is delegated to MAS** -- Synapse no longer serves /login. ├── [COMPOSE] **MAS / Matrix Authentication Service (:8020 web, :8021 health) + mas-db (postgres:16-alpine, LOCAL /var/lib/mas-db)** -- OAuth2/OIDC provider for Matrix. Public: auth.hobbs.farm. Deployed 2026-08-20 to enable QR sign-in (MSC4108). QR sign-in **verified working end-to-end 2026-08-20**. **Every Matrix login now depends on this service.** ├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db) ├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops]) -- **OpenClaw 2026.7.1-2, images PINNED via Dockerfile ARG** (`compose/hf-pdocker-01/ai-agents/{margaret,spirit}/Dockerfile`). Both primary `anthropic/claude-sonnet-4-6`, fallback `google/gemini-2.5-flash`. -├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090) +├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090) -- `traefik:v3.7` (3.7.16, bumped from v3.6 2026-10-26; 3.7 migration notes are k8s-only for us) ├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192. ├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094) ├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server @@ -39,37 +39,39 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha ├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm. ├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310. ├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04 -├── [COMPOSE] **Onyx (RAG/search) -- 10 containers, `/mnt/docker-data/compose/hf-pdocker-01/onyx`.** NOT decommissioned (the bart entry below refers to an older, separate deployment). Public: **onyx.hobbs.farm** -> nginx :3005. Custom build: `gitea_patch.py` + `unstructured_patch.py` index Gitea (`git.hobbs.farm/api/v1`) and call unstructured-api on .100:8000. LLM -> `192.168.86.150:18020/v1` (`qwen3.8-27b`) — **dead while .150 is down.** `onyx-opensearch-1` **stopped 2026-09-21** (`ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false`, held no Onyx data, nothing `depends_on` it; freed ~1.1GB). Restart with `docker compose up -d opensearch` if the flag is ever flipped. +├── [COMPOSE] **Onyx (RAG/search) -- 10 containers, `/mnt/docker-data/compose/hf-pdocker-01/onyx`.** NOT decommissioned (the bart entry below refers to an older, separate deployment). Public: **onyx.hobbs.farm** -> nginx :3005. Custom build: `gitea_patch.py` + `unstructured_patch.py` index Gitea (`git.hobbs.farm/api/v1`) and call unstructured-api on .100:8000. LLM -> `192.168.86.150:18020/v1` (`qwen3.8-27b`) — **dead while .150 is down.** `onyx-opensearch-1` **stopped 2026-09-21** (`ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false`, held no Onyx data, nothing `depends_on` it; freed ~1.1GB). **2026-10-10: it came back on its own after .192's crash/reboot (1.5GB) — now stopped, `restart=no`, and gated behind compose `profiles: ["opensearch"]` so neither reboots nor `compose up` revive it.** Restart with `docker compose --profile opensearch up -d opensearch` if the flag is ever flipped. ├── [COMPOSE] hermes -- **moved here from .150** (host network, `/mnt/docker-data/compose/hf-pdocker-01/hermes`, pinned `v2026.8.18`). Points at `http://192.168.86.150:18020/v1` model `qwen3.8-27b`; that endpoint is **dead**, so every call burns 200s x3 retries and fails. Note this is a different host:port than the `.150:8081` llamacpp setup documented in the GPU Machine section. ├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15) ├── [COMPOSE] synapse-admin -- Synapse admin UI -├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host +├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26) ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB ├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (4x daily 4am/10am/4pm/10pm, log ~/frigate-restart.log) ├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage) bart (15GB RAM, i5-4570T, HDD) -├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12. +├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, **Seerr** -- replaced Jellyseerr 2026-10-17, `seerr/seerr:v3.5.1`, config LOCAL /var/lib/seerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin on official STABLE 12.x, pinned to the dated tag `jellyfin/jellyfin:12.3.20261023-013714` (12.3.0, 2026-10-26).** History: ran the 2026060106 unstable build from 2026-07-14 because the DB had been migrated to the v12 schema (10.11.x cannot read it); stable 12.0 shipped 2026-09-08, moved to 12.2 on 2026-10-17, 12.3 on 2026-10-26 (DB backup `/var/lib/jellyfin-backup-20261026-pre123`). **Never go below 12.** ├── [REMOVED] Onyx -- this *bart* deployment was decommissioned 2026-07-13. **A different, newer Onyx now runs on hf-pdocker-01 (see above) — 'Onyx is gone' is false.** ├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15. ├── [COMPOSE] Farm Agent (:8889) -├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host +├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26) ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB -├── [COMPOSE] Watchtower +├── [COMPOSE] Watchtower -- `nickfedor/watchtower` (maintained fork; containrrr is archived), runs **twice daily 04:00 + 16:00** (`WATCHTOWER_SCHEDULE=0 0 4,16 * * *`) hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD) -├── [REMOVED] UniFi OS Server -- the `unifi-os-server` container (ghcr.io/lemker/unifi-os-server, v5.1.15) was decommissioned 2026-10-05. It was a **duplicate/idle** controller -- the real UniFi controller is the **Podman UniFi OS Server on the Windows box (.150)**, which manages all UniFi devices (e.g. `downstairsswitch` US-24-250W at 192.168.86.159, inform `http://192.168.86.150:8080/inform`). Container + image + network removed; ports 11443/8080/3478/etc freed on .100. **Data preserved** at `/var/lib/unifi-os/*` (971M local) + `/mnt/docker-data/configs/unifi-os/persistent` (NAS); compose dir kept for reference. **Only .150 manages UniFi, by design.** +├── [REMOVED] UniFi OS Server -- the `unifi-os-server` container (ghcr.io/lemker/unifi-os-server, v5.1.15) was decommissioned 2026-10-05. It was a **duplicate/idle** controller -- the real UniFi controller is the **Podman UniFi OS Server on the Windows box (.150)**, which manages all UniFi devices (e.g. `downstairsswitch` US-24-250W at 192.168.86.159, inform `http://192.168.86.150:8080/inform`). Container + image + network removed; ports 11443/8080/3478/etc freed on .100. **Data preserved** at `/var/lib/unifi-os/*` (971M local) + `/mnt/docker-data/configs/unifi-os/persistent` (NAS); compose dir kept for reference. **Only .150 manages UniFi, by design.** **2026-10-26: it was found RUNNING again** (the 10-05 removal never stuck -- journal shows it up again later that same day, then recreated from its compose dir 2026-10-17 21:06Z). Removed again (container, `unifi_default` network, `ghcr.io/lemker/unifi-os-server` + stale `linuxserver/unifi-network-application` images) and the compose file renamed to `docker-compose.yml.decommissioned-20261026` + `DECOMMISSIONED.txt` so a stray `compose up` cannot revive it. Data still preserved. ├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop) ├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13) ├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL) ├── [COMPOSE] Vaultwarden, MCPO, Trivy +├── [COMPOSE] SearXNG (`searxng/searxng:latest`) -- found running 2026-10-26, previously undocumented here +├── [COMPOSE] unstructured-api (:8000, `downloads.unstructured.io/unstructured-io/unstructured-api:latest`) -- used by Onyx's `unstructured_patch.py` on .192 ├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget) ├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation ├── [COMPOSE] CouchDB (:5984) -- Obsidian LiveSync backend for the Second Brain vault. LAN-only. LOCAL disk (NOT NFS). ├── [COMPOSE] Farm Agent (:8889) -├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host +├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host, **v1.12.1** (was v1.7.0 from 2023 until 2026-10-26) ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB -├── [COMPOSE] Watchtower +├── [COMPOSE] Watchtower -- `nickfedor/watchtower` (maintained fork; containrrr is archived), runs **twice daily 04:00 + 16:00** (`WATCHTOWER_SCHEDULE=0 0 4,16 * * *`) ``` @@ -244,7 +246,7 @@ After ANY infrastructure change (deploy, remove, reconfigure, add service, chang 2. **MEMORY.md** — Add/update relevant memory entries 3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus 4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background) -5. **Farm Manager links page** (last audited 2026-08-20: dropped dead Clippy/Bob/Aaron links, added FRIDAY Voice + LinkedStorm + Rongorongo and a new `GPU Machine (192.168.86.150)` category; `hermes` intentionally absent — it publishes no ports. Served path is `/app.js`, not `/static/app.js`.) — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted) +5. **Farm Manager links page** (last audited **2026-10-26**: every link probed; removed both FRIDAY Voice links, added Onyx (public + :3005), Guacamole :8199, Synapse Admin :8080, SearXNG :8888, n8n :5678, CouchDB Fauxton; MCPO -> `/docs`, Trivy -> `/healthz` (API only, no UI); Jellyseerr -> Seerr. Groups live in `/mnt/docker-data/configs/farm-manager/groups.json` -- **update them too when containers are added/renamed**. Previous audit 2026-08-20: dropped dead Clippy/Bob/Aaron links, added FRIDAY Voice + LinkedStorm + Rongorongo and a new `GPU Machine (192.168.86.150)` category; `hermes` intentionally absent — it publishes no ports. Served path is `/app.js`, not `/static/app.js`.) — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted) Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented. @@ -305,7 +307,7 @@ All persistent data on NAS at `/mnt/docker-data/`: 2. **Telegraf Docker Input**: Disabled due to Docker socket permission issues across nodes 3. **Watchtower API Version**: Requires `DOCKER_API_VERSION=1.44` 4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running -5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network) +5. **Frigate cameras**: 192.168.44.0/24 is the **"Camera" VLAN 200** on the Firewalla (gateway 192.168.86.1 / 44.1). Cameras with 44.x IPs MUST sit on Camera-native switch ports. **2026-10-08:** house_north/.51, house_east/.58, house_south/.53, house_west/.52 are wired to downstairsswitch (.159) ports 3/4/18/19 (verified from the switch MAC table 2026-10-11), which are **VLAN 1**, so they can't be reached by routing. They only half-worked because .192's Wi-Fi NIC held a wrong-VLAN 44.x lease. The fix is to make those ports Camera-native (see memory `unifi-estate`). The nodes' `wlp2s0` Wi-Fi NICs must hold **86.x** leases; a 44.x lease there is stale and breaks routing to the real Camera VLAN. **2026-10-10: with the Wi-Fi workaround gone (86.x lease since 10-08), those 4 cameras are now fully DOWN (0 fps, no ping) until the ports are re-profiled.** 6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate 7. **systemd-resolved DNSStubListener**: disabled on hf-pdocker-01 and hf-pdocker-02 via `/etc/systemd/resolved.conf.d/nostub.conf` (`DNSStubListener=no`) — originally for AdGuard Home's port 53. AdGuard was decommissioned 2026-07-13; the stub-listener change is harmless and left in place (re-enable only if something needs 127.0.0.53). 8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`. @@ -327,7 +329,7 @@ All persistent data on NAS at `/mnt/docker-data/`: 23. **Matrix registration was wide open to the internet (fixed 2026-08-20)**: `mess.hobbs.farm` is publicly reachable (Traefik, port 443, federation + client API on :8008) and `homeserver.yaml` had `enable_registration: true` + `enable_registration_without_verification: true` — the registration flow was a bare `m.login.dummy`, i.e. **anyone on the internet could create an account on hobbs.farm with no email, captcha, or token**. No abuse had occurred (all 18 accounts are known/expected; newest legit account was @fry 2026-07-05). Fixed by setting both to `false` and `restrict_public_rooms_to_local_users: true` (was `false`, which published the public room directory to any remote server). Backup: `/var/lib/synapse/homeserver.yaml.bak-20260820-registration`. New accounts are still creatable on the box via `docker exec synapse register_new_matrix_user -k `. 24. **Synapse database history (SQLite -> local disk -> PostgreSQL, all 2026-08-20)**: Synapse ran on **SQLite on NFS** while CLAUDE.md claimed it used a `synapse-db` postgres that in fact had **0 tables**. First the data was moved to LOCAL `/var/lib/synapse` and the empty postgres deleted; then, because `syn2mas` **only reads PostgreSQL** ("The migration tool only supports reading from PostgreSQL for the Synapse database as well"), a real `synapse-db` was created and the DB ported with `synapse_port_db` (20 users / 93,888 events / 65 devices, verified identical; 2m35s downtime). The PG database MUST have `LC_COLLATE=C` -- verify with `select datcollate from pg_database where datname='synapse'` (PG16 removed the `SHOW lc_collate` GUC). The old SQLite file is still at `/var/lib/synapse/homeserver.db`, now inert; NFS copy at `/mnt/docker-data/configs/matrix/data` is an older backup -- **never remount either**. -25. **Synapse upgraded 1.148.0 -> 1.159.0 and moved to the element-hq registry (2026-08-20)**: it had been on `matrixdotorg/synapse:latest`, an **abandoned Docker Hub repo** whose `:latest` froze at 1.148.0 (image built 2026-02-24) — a moving tag that silently stopped moving on a publicly-exposed homeserver. Now pinned to `ghcr.io/element-hq/synapse:v1.159.0` (note the **`v` prefix** — `:1.159.0` without it 404s). Upgrade was clean: schema 93 -> 94, all background updates finished in ~5s, 0 errors, 19 users / 93,884 events intact, memory 112MB/1G. None of the v1.149-v1.159 breaking changes applied to us (MSC3861 removal — unused; `quarantined_media_changes` stream writer — workers only; the rest are Debian/Ubuntu packaging). Pre-upgrade backup: `/var/lib/synapse.bak-20260820-pre1159` (646MB) — delete once you're satisfied. **Pinned on purpose: bump it manually and deliberately, since the last moving tag rotted for 6 months unnoticed.** +25. **Synapse upgraded 1.148.0 -> 1.159.0 and moved to the element-hq registry (2026-08-20)**: it had been on `matrixdotorg/synapse:latest`, an **abandoned Docker Hub repo** whose `:latest` froze at 1.148.0 (image built 2026-02-24) — a moving tag that silently stopped moving on a publicly-exposed homeserver. Now pinned to `ghcr.io/element-hq/synapse:v1.159.0` (note the **`v` prefix** — `:1.159.0` without it 404s). Upgrade was clean: schema 93 -> 94, all background updates finished in ~5s, 0 errors, 19 users / 93,884 events intact, memory 112MB/1G. None of the v1.149-v1.159 breaking changes applied to us (MSC3861 removal — unused; `quarantined_media_changes` stream writer — workers only; the rest are Debian/Ubuntu packaging). Pre-upgrade backup: `/var/lib/synapse.bak-20260820-pre1159` (646MB) — delete once you're satisfied. **Pinned on purpose: bump it manually and deliberately, since the last moving tag rotted for 6 months unnoticed.** **2026-10-26: bumped v1.159.0 -> v1.163.2** (schema 94 deltas applied in <1s; config contains none of the options affected by the 1.160-1.163 upgrade notes). pg_dump backups taken first: `/var/lib/upgrade-backups-20261026/{synapse-db,mas-db}.dump` on .192. MAS bumped 1.23.0 -> **1.27.1** at the same time. 26. **`SYNAPSE_SERVER_NAME` in the matrix compose disagrees with homeserver.yaml**: compose passes `SYNAPSE_SERVER_NAME=mess.hobbs.farm` but the real `server_name` is **`hobbs.farm`** (user IDs are `@user:hobbs.farm`). The env var is only read when the image *generates* a config, so it's inert today — but if `homeserver.yaml` ever went missing, the container would generate a fresh config with the wrong `server_name` AND a new signing key, which would break federation and every existing user ID. Left as-is (changing it has no effect while the config exists); just never let the container auto-generate.. 27. **Matrix auth is delegated to MAS -- Synapse does NOT serve /login any more (2026-08-20)**: with `matrix_authentication_service.enabled: true`, Synapse answers `M_UNRECOGNIZED` on `/_matrix/client/*/login`. The **reverse proxy must route the three compat paths to MAS or nobody can sign in**: `login`, `logout`, `refresh` (verified: MAS returns 200/405 on those and 404 on `register`/`whoami`, which stay on Synapse). Implemented as the `mess-auth-compat` router in `/mnt/docker-data/traefik/dynamic/temp-routes.yml` with explicit `priority: 1000` so it beats the catch-all `mess` router. **Anything that talks to `192.168.86.192:8008` directly bypasses Traefik and therefore cannot log in with a password** -- token-based bots (spirit, frigate, alertbot) are unaffected because they never call /login, but `@clippyauto` (n8n) and `@pluto` historically re-login with a password (17 and 15 accumulated devices). They currently run on migrated tokens; if either ever needs a fresh password login, point it at `https://mess.hobbs.farm` instead of `:8008`. @@ -343,3 +345,11 @@ All persistent data on NAS at `/mnt/docker-data/`: 34. **Onyx self-registration was open to the internet (fixed 2026-09-21)**: `onyx.hobbs.farm` is publicly reachable and Onyx runs `AUTH_TYPE=basic` with **no** `REQUIRE_EMAIL_VERIFICATION` and **no** `VALID_EMAIL_DOMAINS` — so `POST /api/auth/register` let anyone on the internet create an account. External scanners were already probing the host (a DigitalOcean IP hit `/` on 2026-09-21). No abuse had occurred: only 2 accounts existed (`glyptic@gmail.com` + Onyx's internal `anonymous@onyx.app`). **Fix: a Traefik router `onyx-register-lockdown` (`priority: 1000`, beats the catch-all `onyx` router) matching `PathPrefix(/api/auth/register)` behind an `ipAllowList` middleware limited to `192.168.86.0/24` + `127.0.0.1/32`.** Verified by temporarily narrowing the range: register returned **403** from a non-allowlisted source while `/api/auth/login` (405) and `/` (200) were unaffected; range then restored. Config `/mnt/docker-data/traefik/dynamic/onyx.yml` (backup `.bak-20260920-195851`). **To add a user: register from the LAN, or widen the `sourceRange`.** Note `VALID_EMAIL_DOMAINS` was rejected as the control because it is domain-level only — `gmail.com` would admit any Gmail user. Same class as Known Issue #23 (open Matrix registration). 35. **The live Prometheus config is `/mnt/docker-data/configs/prometheus/prometheus.yml` — the copy at `compose/hf-pdocker-01/monitoring/prometheus.yml` is STALE and misleading (2026-09-21)**: the compose-dir copy still lists decommissioned targets, including Onyx at the wrong node (`192.168.86.167:3005`, bart) and OpenVAS (`:9392`). Neither is in the live config, so neither was ever probed — grepping the compose copy to answer "is X monitored?" gives the wrong answer. Onyx had **no probe at all** until one was added to the live file on 2026-09-21 (`http://192.168.86.192:3005`, now `probe_success=1`). Backup `prometheus.yml.bak-20260920-*`. The live file is also the better changelog: it records the voicecoach (2026-08-24) and FRIDAY Voice / speaches (2026-08-26) teardowns that CLAUDE.md still described as live. 36. **SSH key auth to hf-pdocker-01 is broken — password only (2026-09-21)**: all of `~/.ssh/id_docker_server`, `id_bart`, `id_nas` are rejected by `192.168.86.192` (`Permission denied (publickey,password)`), while `id_docker_server` still works for `.100` and `id_bart` for bart. Connect with `sshpass -p '&Rip8a115' ssh -o PubkeyAuthentication=no glyptic@192.168.86.192`. **Any key-based automation targeting .192 is silently failing** — check `~/.ssh/authorized_keys` on that node (likely lost when the host was rebuilt/restored). +37. **hf-pdocker-01 hard-hung for ~27h (2026-10-09 13:55 → 2026-10-10 17:08 CDT)**: journal simply stops at 13:55 with no OOM, kernel error, or shutdown record; `sar` at 13:50 showed a healthy box (62% mem, 23% swap, load ~4), and .100/bart stayed up (9d uptime), so it was neither memory exhaustion nor a house-wide power event. Came back on kernel 7.0.0-38 (was -34) after a power cycle. Cause unknown — if it recurs, suspect hardware (this box was repaired 2026-03) and consider enabling persistent `kdump`/`netconsole`. Side effects found on 2026-10-10: `onyx-opensearch-1` revived itself (fixed, see Onyx entry). Also fixed then: `ServiceHighMemory` used `container_memory_usage_bytes`, which counts reclaimable page cache, so clip-service false-alarmed at 94% while really at 64% — rule now uses `container_memory_working_set_bytes` (backup `alerts.yml.bak-20261010-workingset`). +38. **bart shows ~20 nameless `Dead` containers in `docker ps -a`** — orphaned `/var/lib/docker/containers/` dirs for containers whose images Watchtower already deleted. `docker rm` says "No such container". Harmless clutter; clearing them requires stopping dockerd (moving the dirs aside) which restarts the whole media stack, so it was left for a maintenance window. +39. **[RESOLVED 2026-10-11] Static-IP ESPHome boards were invisible behind UniFi Proxy ARP (found 2026-10-10)**: the `ASUS_F0_2G` WLAN has **Proxy ARP on** (`proxy_arp=1` in every AP's `/etc/hostapd/wifi0ap0.cfg`; it also has `ap_isolate=1` and `mcast.enhance=1`). With Proxy ARP, the AP answers ARP for wireless clients **from a table it learns by snooping DHCP** and stops passing ARP broadcasts through to them. Boards with ESPHome `manual_ip` never DHCP, so after any reconnect nobody answers ARP for them: they associate fine and even announce themselves over mDNS, but are unreachable. **Pool Value Controls (.6, MAC `a4:cf:12:f1:6e:c8`) went dark 2026-10-09 11:04** at a Wi-Fi reconnect on AP .187 and then looped every 15 min on ESPHome's API `reboot_timeout`. Diagnosed by capture on .187: ARP who-has .6 reached the radio and got no reply, while a static neighbor entry made ping and API work at once. **Workaround (live):** systemd unit `static-arp-esphome.service` on .192 pins `.6 -> a4:cf:12:f1:6e:c8 nud permanent` on `enp0s31f6` (HA and ESPHome are bridge-networked on .192, so they use the host's neighbor table). **Real fix (pending, needs controller access on .150):** turn Proxy ARP off on the `ASUS_F0_2G` WLAN, OR switch the boards to DHCP + Firewalla reservations. Once Proxy ARP is off, `systemctl disable --now static-arp-esphome` on .192. A `power_save_mode: none` change was also OTA'd to the board (backup `pool-value-controls.yaml.bak-20261010-powersave`). It did NOT fix this, but it's harmless and was left in. **Any other static-IP Wi-Fi device on that SSID has the same problem.** **RESOLVED 2026-10-11:** Proxy ARP (and `ap_isolate`) were turned off on `ASUS_F0_2G` in the controller (pushed to all APs 11:09–11:18; the `proxy_arp`/`ap_isolate` lines are gone from every AP's hostapd config). `static-arp-esphome.service` was then disabled and deleted on .192. Verified afterwards: .6 resolves by normal ARP (neighbor `REACHABLE`, not `PERMANENT`), ping OK, HA API session established. If a static-IP Wi-Fi device goes dark again, check that Proxy ARP hasn't been switched back on. polelightpond (.14) and living_room_lights (.153) were still unreachable after the fix, so they have some other problem (unpowered or offline). + +40. **Container version audit + upgrade round (2026-10-26)**. Upgraded: Synapse v1.163.2, MAS 1.27.1, Traefik v3.7 (3.7.16), node-exporter v1.12.1 (all 3 nodes), cadvisor v0.60.6, matrix-alertmanager-receiver 2026.9.23, Gitea 1.26.4 (patch only; pg_dump at `~/gitea-db-20261026-pre1264.dump` on .100), Jellyfin 12.3, Seerr v3.5.1, plus re-pulls of every moving tag on .192 (HA 2026.10.4, ESPHome, Mosquitto, Prometheus, Blackbox, Telegraf, nginx, litellm, and the postgres/mysql/redis patch images). Every compose file touched has a `.bak-20261026` next to it on the NAS. **Deliberately NOT done, each needs its own decision:** Onyx `:latest` now = **v5.0.0 (major, released 2026-10-24)**, still on v4.9.2 (no Watchtower on .192, so it only moves on a manual pull); Gitea 1.27.x / **28.x** (major jumps); OpenClaw 2026.7.1-2 -> 2026.10.7 (one-way state migration, see #30); hermes v2026.8.18 -> v0.22.5 (upstream switched to semver tags, and its LLM endpoint is dead anyway); **MySQL 8.0 (guacamole-db, semaphore-db) went EOL April 2026 -> needs an 8.4 LTS migration with a dump/restore**; InfluxDB 1.8 (EOL); Onyx's `postgres:15.2-alpine` (2023 patch level, managed by Onyx's compose); `awesometechnologies/synapse-admin` (stale, maintained fork is `etkecc/synapse-admin`). How to audit: compare each container's `RepoDigests` against registry HEAD digests -- for Docker Hub use the hub.docker.com web API (`/v2/namespaces//repositories//tags/` -> `digest`), which does not count against pull limits. `docker buildx imagetools inspect --format` is ignored by the nodes' buildx v0.31.1. +41. **Docker Hub anonymous pull limit is shared by the whole house** (public IP 98.160.103.126, 100 pulls/hour): on 2026-10-26 every node AND WSL got 429 `toomanyrequests`, meaning Watchtower updates of Docker Hub images were silently failing. Mitigation applied: both Watchtowers moved from a 24h poll interval to a **twice-daily cron schedule** (04:00/16:00), and the archived `containrrr/watchtower` swapped for the maintained drop-in fork `nickfedor/watchtower` (**image name is `nickfedor/`, not `nicholas-fedor/`** -- the latter is the GitHub org and 404s on Hub). Same env vars and still honors `com.centurylinklabs.watchtower.enable`. A `docker login` on the nodes would raise the limit further. **cadvisor no longer publishes to gcr.io** after v0.55 -- the image is now `ghcr.io/google/cadvisor`. +42. **node-exporter had been launched from `/tmp` (compose project `tmp`) on all 3 nodes**, so `docker compose` in the real dirs didn't own it. Fixed 2026-10-26: removed and redeployed from `/mnt/docker-data/compose//node-exporter/`. Separately, the **monitoring** compose on .192 still defines a legacy `node-exporter:latest` service on :9100 that a bare `docker compose up -d` would try to start; it is now gated behind `profiles: ["legacy-node-exporter"]`. Also: the matrix-alertmanager-receiver container is now named `monitoring-matrix-alertmanager-receiver-1` (compose default) -- Alertmanager reaches it by service name `matrix-alertmanager-receiver`, which still resolves. +43. **Farm Manager showed ZERO containers for hf-pdocker-01 (fixed 2026-10-26)**: the server fetches each agent's `/containers` with a timeout and on any error silently returns `[]` for that node. .192's agent took ~12s (47 containers, per-container `stats(stream=False)` = Docker samples twice ~1s apart, only 10 workers) vs a 10s timeout, so the main host's containers simply vanished from the dashboard while `/api/nodes` still said "healthy, 47". Fix: agent uses `stats(..., one_shot=True)` + 20 workers (now ~2s), server timeout 30s. Agent rebuilt on .192 and shipped to .100/bart via `docker save | docker load` (only .192 has the `/opt/farm-manager` source); rollback tags `farm-agent:rollback-20261026` / `farm-manager:rollback-20261026`. Same day: removed a dead `gpu-workstation` node (192.168.86.27) from the live `config.json`, made `/health` ignore Dead containers (bart reported 30 instead of 10), and rebuilt `groups.json` -- every group still used Swarm-era / decommissioned names (`ai-agents_margaret`, `jellyseerr`, `adguardhome`, bart Onyx, `openvas`, `unifi`...), so all group actions were no-ops. Now 14 groups / 63 containers; per-node utilities (farm-agent, node-exporter, telegraf, watchtower) are deliberately ungrouped so a group "stop" can't kill the agents. **4 tests in `tests/test_agent_docker_ops.py` fail -- pre-existing** (they mock `containers.list` but the code uses low-level `api.containers`); not caused by this change. diff --git a/compose/bart/media/docker-compose.yml b/compose/bart/media/docker-compose.yml index 9b83b11..13c534a 100644 --- a/compose/bart/media/docker-compose.yml +++ b/compose/bart/media/docker-compose.yml @@ -1,11 +1,10 @@ services: jellyfin: - # PINNED to the 12.0.0 unstable build that migrated the DB to the v12 - # schema (DropExtraIdsColumn) — :latest is still 10.11.x and CANNOT read - # this DB ("no such column: b.ExtraIds"). Do not unpin until official - # stable >= 12 ships. (2026-07-14: a :latest deploy downgraded it and - # broke Resume/home sections; repinned.) - image: jellyfin/jellyfin:2026060106 + # Pinned to official stable 12.2. History: ran the 2026060106 unstable + # build from 2026-07-14 because the DB had been migrated to the v12 schema + # and 10.11.x could not read it. Stable 12 shipped 2026-09; moved to + # 12.2 on 2026-10-17, 12.3 on 2026-10-26. Never go back below 12. + image: jellyfin/jellyfin:12.3.20261023-013714 container_name: jellyfin restart: unless-stopped environment: @@ -145,14 +144,19 @@ services: max-size: "50m" max-file: "3" - jellyseerr: - image: fallenbagel/jellyseerr:latest - container_name: jellyseerr + seerr: + # Seerr = merged successor of Jellyseerr/Overseerr (fallenbagel/jellyseerr + # is deprecated). Migrated 2026-10-17; DB auto-migrated on first start. + # Runs as node (UID 1000), needs init. LOCAL disk (SQLite). Old config left + # at /mnt/docker-data/configs/jellyseerr for rollback. + image: seerr/seerr:v3.5.1 + init: true + container_name: seerr restart: unless-stopped environment: - TZ=America/Chicago volumes: - - /mnt/docker-data/configs/jellyseerr:/app/config + - /var/lib/seerr:/app/config ports: - "5055:5055" mem_limit: 512M diff --git a/compose/bart/node-exporter/docker-compose.yml b/compose/bart/node-exporter/docker-compose.yml index 44c01a5..cc1c1b6 100644 --- a/compose/bart/node-exporter/docker-compose.yml +++ b/compose/bart/node-exporter/docker-compose.yml @@ -1,6 +1,6 @@ services: node-exporter: - image: prom/node-exporter:v1.7.0 + image: prom/node-exporter:v1.12.1 container_name: node-exporter hostname: bart network_mode: host @@ -11,14 +11,14 @@ services: - /sys:/host/sys:ro - /:/rootfs:ro command: - - '--path.procfs=/host/proc' - - '--path.rootfs=/rootfs' - - '--path.sysfs=/host/sys' - - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' + - "--path.procfs=/host/proc" + - "--path.rootfs=/rootfs" + - "--path.sysfs=/host/sys" + - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)" logging: - driver: "json-file" + driver: json-file options: - max-size: "50m" + max-size: 50m max-file: "3" labels: - - "com.centurylinklabs.watchtower.enable=false" + - com.centurylinklabs.watchtower.enable=false diff --git a/compose/bart/watchtower/docker-compose.yml b/compose/bart/watchtower/docker-compose.yml index f531d5a..0b1e917 100644 --- a/compose/bart/watchtower/docker-compose.yml +++ b/compose/bart/watchtower/docker-compose.yml @@ -1,6 +1,6 @@ services: watchtower: - image: containrrr/watchtower:latest + image: nickfedor/watchtower:latest # containrrr/watchtower is archived upstream (2026-10-26) container_name: watchtower restart: unless-stopped volumes: @@ -8,7 +8,7 @@ services: environment: - WATCHTOWER_CLEANUP=true - WATCHTOWER_INCLUDE_RESTARTING=true - - WATCHTOWER_POLL_INTERVAL=86400 + - WATCHTOWER_SCHEDULE=0 0 4,16 * * * # twice daily, 04:00 + 16:00 local - WATCHTOWER_MONITOR_ONLY=false - TZ=America/Chicago - DOCKER_API_VERSION=1.44 diff --git a/compose/hf-pdocker-01/mas/docker-compose.yml b/compose/hf-pdocker-01/mas/docker-compose.yml index 4efa43a..478e6f4 100644 --- a/compose/hf-pdocker-01/mas/docker-compose.yml +++ b/compose/hf-pdocker-01/mas/docker-compose.yml @@ -32,7 +32,7 @@ services: # NOTE: MAS tags are NOT v-prefixed (1.23.0), unlike Synapse (v1.159.0). # Distroless image - no shell, so no container healthcheck is possible. # Liveness is covered by the blackbox probe of https://auth.hobbs.farm/health. - image: ghcr.io/element-hq/matrix-authentication-service:1.23.0 + image: ghcr.io/element-hq/matrix-authentication-service:1.27.1 container_name: mas restart: unless-stopped command: ["server", "--config", "/config/config.yaml"] diff --git a/compose/hf-pdocker-01/matrix/docker-compose.yml b/compose/hf-pdocker-01/matrix/docker-compose.yml index e70b5c6..b7b7221 100644 --- a/compose/hf-pdocker-01/matrix/docker-compose.yml +++ b/compose/hf-pdocker-01/matrix/docker-compose.yml @@ -2,7 +2,7 @@ services: synapse: # 2026-08-20: matrixdotorg repo is abandoned (froze at 1.148.0); upstream is now element-hq. # Pinned deliberately so the running version is visible in the file - bump manually. - image: ghcr.io/element-hq/synapse:v1.159.0 + image: ghcr.io/element-hq/synapse:v1.163.2 container_name: synapse restart: unless-stopped environment: diff --git a/compose/hf-pdocker-01/monitoring/docker-compose.yml b/compose/hf-pdocker-01/monitoring/docker-compose.yml index 0552447..626ae1e 100644 --- a/compose/hf-pdocker-01/monitoring/docker-compose.yml +++ b/compose/hf-pdocker-01/monitoring/docker-compose.yml @@ -1,21 +1,12 @@ -networks: - monitoring-net: - name: monitoring-net - services: grafana: image: grafana/grafana:latest - container_name: grafana restart: unless-stopped user: "104" - networks: - - monitoring-net ports: - "3001:3000" volumes: - /mnt/docker-data/configs/grafana:/var/lib/grafana - mem_limit: 512M - memswap_limit: 512M healthcheck: test: ["CMD-SHELL", "wget -q --spider http://localhost:3000/api/health || exit 1"] interval: 30s @@ -27,13 +18,19 @@ services: options: max-size: "50m" max-file: "3" + deploy: + mode: replicated + replicas: 1 + placement: + constraints: + - node.hostname == hf-pdocker-01 + resources: + limits: + memory: 512M prometheus: image: prom/prometheus:latest - container_name: prometheus restart: unless-stopped - networks: - - monitoring-net ports: - "9095:9090" command: @@ -44,8 +41,6 @@ services: - '--web.enable-lifecycle' volumes: - /mnt/docker-data/configs/prometheus:/etc/prometheus - mem_limit: 512M - memswap_limit: 512M healthcheck: test: ["CMD-SHELL", "wget -q --spider http://localhost:9090/-/healthy || exit 1"] interval: 30s @@ -57,13 +52,19 @@ services: options: max-size: "50m" max-file: "3" + deploy: + mode: replicated + replicas: 1 + placement: + constraints: + - node.hostname == hf-pdocker-01 + resources: + limits: + memory: 512M alertmanager: image: prom/alertmanager:latest - container_name: alertmanager restart: unless-stopped - networks: - - monitoring-net ports: - "9093:9093" command: @@ -72,8 +73,6 @@ services: volumes: - /mnt/docker-data/configs/alertmanager:/etc/alertmanager - alertmanager-data:/alertmanager - mem_limit: 256M - memswap_limit: 256M healthcheck: test: ["CMD-SHELL", "wget -q --spider http://localhost:9093/-/healthy || exit 1"] interval: 30s @@ -85,19 +84,49 @@ services: options: max-size: "50m" max-file: "3" + deploy: + mode: replicated + replicas: 1 + placement: + constraints: + - node.hostname == hf-pdocker-01 + resources: + limits: + memory: 256M + + node-exporter: + # DISABLED 2026-10-26: node-exporter runs STANDALONE (compose//node-exporter, pid:host). + # This legacy copy would collide on :9100 if started; gated behind a profile so a + # bare `docker compose up -d` here never starts it. + profiles: ["legacy-node-exporter"] + image: prom/node-exporter:latest + restart: unless-stopped + ports: + - "9100:9100" + volumes: + - /proc:/host/proc:ro + - /sys:/host/sys:ro + - /:/rootfs:ro + command: + - '--path.procfs=/host/proc' + - '--path.rootfs=/rootfs' + - '--path.sysfs=/host/sys' + logging: + driver: "json-file" + options: + max-size: "50m" + max-file: "3" + deploy: + mode: global + resources: + limits: + memory: 128M influxdb: image: influxdb:1.8-alpine - container_name: influxdb restart: unless-stopped - networks: - - monitoring-net - ports: - - "8086:8086" volumes: - /mnt/docker-data/configs/influxdb:/var/lib/influxdb - mem_limit: 1G - memswap_limit: 1G healthcheck: test: ["CMD-SHELL", "wget -q --spider http://localhost:8086/ping || exit 1"] interval: 30s @@ -109,45 +138,99 @@ services: options: max-size: "50m" max-file: "3" + deploy: + mode: replicated + replicas: 1 + placement: + constraints: + - node.hostname == hf-pdocker-01 + resources: + limits: + memory: 1G matrix-alertmanager-receiver: - image: metio/matrix-alertmanager-receiver:2026.2.25 - container_name: matrix-alertmanager-receiver + image: metio/matrix-alertmanager-receiver:2026.9.23 restart: unless-stopped - networks: - - monitoring-net ports: - "9094:9094" volumes: - /mnt/docker-data/configs/alertmanager/matrix-receiver-config.yml:/config/config.yml:ro command: ["--config-path", "/config/config.yml"] - mem_limit: 128M - memswap_limit: 128M + # NOTE: No healthcheck - this image is distroless (no shell), so CMD-SHELL healthchecks fail. + # The service is verified healthy by Alertmanager successfully POSTing to it. logging: driver: "json-file" options: max-size: "50m" max-file: "3" + deploy: + mode: replicated + replicas: 1 + placement: + constraints: + - node.hostname == hf-pdocker-01 + resources: + limits: + memory: 128M + + cadvisor: + image: ghcr.io/google/cadvisor:v0.60.6 # moved off gcr.io (no releases there after v0.55) + restart: unless-stopped + privileged: true + command: + - '--docker_only=true' + - '--disable_metrics=advtcp,cpu_topology,cpuset,hugetlb,memory_numa,process,referenced_memory,resctrl,sched,tcp,udp,disk,diskIO,percpu' + - '--housekeeping_interval=30s' + - '--store_container_labels=false' + ports: + - "8081:8080" + volumes: + - /:/rootfs:ro + - /var/run:/var/run:ro + - /sys:/sys:ro + - /var/lib/docker/:/var/lib/docker:ro + - /dev/disk/:/dev/disk:ro + - /sys/fs/cgroup:/sys/fs/cgroup:ro + devices: + - /dev/kmsg + logging: + driver: "json-file" + options: + max-size: "50m" + max-file: "3" + deploy: + mode: replicated + replicas: 1 + placement: + constraints: + - node.hostname == hf-pdocker-01 + resources: + limits: + memory: 256M blackbox-exporter: image: prom/blackbox-exporter:latest - container_name: blackbox-exporter restart: unless-stopped - networks: - - monitoring-net ports: - "9115:9115" command: - '--config.file=/config/blackbox.yml' volumes: - /mnt/docker-data/configs/blackbox:/config:ro - mem_limit: 128M - memswap_limit: 128M logging: driver: "json-file" options: max-size: "50m" max-file: "3" + deploy: + mode: replicated + replicas: 1 + placement: + constraints: + - node.hostname == hf-pdocker-01 + resources: + limits: + memory: 128M volumes: alertmanager-data: diff --git a/compose/hf-pdocker-01/node-exporter/docker-compose.yml b/compose/hf-pdocker-01/node-exporter/docker-compose.yml index 1a2310d..218c2b7 100644 --- a/compose/hf-pdocker-01/node-exporter/docker-compose.yml +++ b/compose/hf-pdocker-01/node-exporter/docker-compose.yml @@ -1,6 +1,6 @@ services: node-exporter: - image: prom/node-exporter:v1.7.0 + image: prom/node-exporter:v1.12.1 container_name: node-exporter hostname: hf-pdocker-01 network_mode: host diff --git a/compose/hf-pdocker-01/traefik/docker-compose.yml b/compose/hf-pdocker-01/traefik/docker-compose.yml index b57349c..a605633 100644 --- a/compose/hf-pdocker-01/traefik/docker-compose.yml +++ b/compose/hf-pdocker-01/traefik/docker-compose.yml @@ -1,6 +1,6 @@ services: traefik: - image: traefik:v3.6 + image: traefik:v3.7 container_name: traefik restart: unless-stopped command: diff --git a/compose/hf-pdocker-02/gitea/docker-compose.yml b/compose/hf-pdocker-02/gitea/docker-compose.yml index 7ad2155..b138695 100644 --- a/compose/hf-pdocker-02/gitea/docker-compose.yml +++ b/compose/hf-pdocker-02/gitea/docker-compose.yml @@ -1,6 +1,6 @@ services: gitea: - image: gitea/gitea:1.26.1 + image: gitea/gitea:1.26.4 container_name: gitea restart: unless-stopped labels: diff --git a/compose/hf-pdocker-02/node-exporter/docker-compose.yml b/compose/hf-pdocker-02/node-exporter/docker-compose.yml index 2c5575f..0dd8a92 100644 --- a/compose/hf-pdocker-02/node-exporter/docker-compose.yml +++ b/compose/hf-pdocker-02/node-exporter/docker-compose.yml @@ -1,6 +1,6 @@ services: node-exporter: - image: prom/node-exporter:v1.7.0 + image: prom/node-exporter:v1.12.1 container_name: node-exporter hostname: hf-pdocker-02 network_mode: host @@ -11,14 +11,14 @@ services: - /sys:/host/sys:ro - /:/rootfs:ro command: - - '--path.procfs=/host/proc' - - '--path.rootfs=/rootfs' - - '--path.sysfs=/host/sys' - - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' + - "--path.procfs=/host/proc" + - "--path.rootfs=/rootfs" + - "--path.sysfs=/host/sys" + - "--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)" logging: - driver: "json-file" + driver: json-file options: - max-size: "50m" + max-size: 50m max-file: "3" labels: - - "com.centurylinklabs.watchtower.enable=false" + - com.centurylinklabs.watchtower.enable=false diff --git a/compose/hf-pdocker-02/unifi/DECOMMISSIONED.txt b/compose/hf-pdocker-02/unifi/DECOMMISSIONED.txt new file mode 100644 index 0000000..de0de7d --- /dev/null +++ b/compose/hf-pdocker-02/unifi/DECOMMISSIONED.txt @@ -0,0 +1,3 @@ +Decommissioned 2026-10-05 and again 2026-10-26 (it had been revived and was running). +Duplicate controller; the real UniFi controller is the Podman UniFi OS Server on .150. +Do NOT compose up. Data kept in /var/lib/unifi-os (.100) + /mnt/docker-data/configs/unifi-os/persistent. diff --git a/compose/hf-pdocker-02/unifi/docker-compose.yml b/compose/hf-pdocker-02/unifi/docker-compose.yml.decommissioned-20261026 similarity index 100% rename from compose/hf-pdocker-02/unifi/docker-compose.yml rename to compose/hf-pdocker-02/unifi/docker-compose.yml.decommissioned-20261026 diff --git a/compose/hf-pdocker-02/watchtower/docker-compose.yml b/compose/hf-pdocker-02/watchtower/docker-compose.yml index 48b3d78..0b1e917 100644 --- a/compose/hf-pdocker-02/watchtower/docker-compose.yml +++ b/compose/hf-pdocker-02/watchtower/docker-compose.yml @@ -1,6 +1,6 @@ services: watchtower: - image: containrrr/watchtower:latest + image: nickfedor/watchtower:latest # containrrr/watchtower is archived upstream (2026-10-26) container_name: watchtower restart: unless-stopped volumes: @@ -8,12 +8,11 @@ services: environment: - WATCHTOWER_CLEANUP=true - WATCHTOWER_INCLUDE_RESTARTING=true - - WATCHTOWER_POLL_INTERVAL=86400 + - WATCHTOWER_SCHEDULE=0 0 4,16 * * * # twice daily, 04:00 + 16:00 local - WATCHTOWER_MONITOR_ONLY=false - TZ=America/Chicago - DOCKER_API_VERSION=1.44 mem_limit: 256M - memswap_limit: 256M logging: driver: "json-file" options: