fix: nzbhydra2 stale NFS H2 lock -> local /var/lib/nzbhydra; add Farm Manager links page to post-change checklist

nzbhydra2 was the 4th victim of the 2026-07-12 power-event stale NFS locks
(H2 'file is locked' retry loop; the earlier mem bump was a red herring).
Farm Manager links updated (AdGuard/Actual Budget/Portainer removed, UniFi
-> :11443) and image rebuilt; links maintenance is now checklist item 5.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-13 20:39:14 -05:00
parent d35af1ddad
commit 45573cf9fc
2 changed files with 5 additions and 2 deletions
+2 -1
View File
@@ -232,6 +232,7 @@ After ANY infrastructure change (deploy, remove, reconfigure, add service, chang
2. **MEMORY.md** — Add/update relevant memory entries 2. **MEMORY.md** — Add/update relevant memory entries
3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus 3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus
4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background) 4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
5. **Farm Manager links page** — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)
Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented. Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.
@@ -303,7 +304,7 @@ All persistent data on NAS at `/mnt/docker-data/`:
13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`. 13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`. 14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11). 15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr` on bart). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.** 16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`). 17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`).
18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router. 18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router.
19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched. 19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.
+3 -1
View File
@@ -119,7 +119,9 @@ services:
- PGID=1000 - PGID=1000
- TZ=America/Chicago - TZ=America/Chicago
volumes: volumes:
- /mnt/docker-data/configs/nzbhydra:/config # LOCAL disk, not NFS: stale NFS lock on the H2 database after the
# 2026-07-12 power event kept hydra unhealthy in a DB-retry loop
- /var/lib/nzbhydra:/config
ports: ports:
- "5076:5076" - "5076:5076"
mem_limit: 1G mem_limit: 1G