From 45573cf9fcb6d445904665303ec45a1f02c3e596 Mon Sep 17 00:00:00 2001 From: jeremy Date: Mon, 13 Jul 2026 20:39:14 -0500 Subject: [PATCH] fix: nzbhydra2 stale NFS H2 lock -> local /var/lib/nzbhydra; add Farm Manager links page to post-change checklist nzbhydra2 was the 4th victim of the 2026-07-12 power-event stale NFS locks (H2 'file is locked' retry loop; the earlier mem bump was a red herring). Farm Manager links updated (AdGuard/Actual Budget/Portainer removed, UniFi -> :11443) and image rebuilt; links maintenance is now checklist item 5. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 3 ++- compose/bart/media/docker-compose.yml | 4 +++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index fcc432d..a36cdc0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -232,6 +232,7 @@ After ANY infrastructure change (deploy, remove, reconfigure, add service, chang 2. **MEMORY.md** — Add/update relevant memory entries 3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus 4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background) +5. **Farm Manager links page** — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted) Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented. @@ -303,7 +304,7 @@ All persistent data on NAS at `/mnt/docker-data/`: 13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`. 14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`. 15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11). -16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr` on bart). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.** +16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.** 17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`). 18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router. 19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched. diff --git a/compose/bart/media/docker-compose.yml b/compose/bart/media/docker-compose.yml index 6cde4c7..a48ed54 100644 --- a/compose/bart/media/docker-compose.yml +++ b/compose/bart/media/docker-compose.yml @@ -119,7 +119,9 @@ services: - PGID=1000 - TZ=America/Chicago volumes: - - /mnt/docker-data/configs/nzbhydra:/config + # LOCAL disk, not NFS: stale NFS lock on the H2 database after the + # 2026-07-12 power event kept hydra unhealthy in a DB-retry loop + - /var/lib/nzbhydra:/config ports: - "5076:5076" mem_limit: 1G