jeremy/hobbs-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: move jellyfin config to local /var/lib/jellyfin (NFS SQLite lock stalls); decommission OpenVAS

Browse Source 
Jellyfin lockups root-caused to jellyfin.db (WAL mode) on NFS: SQLite
"database is locked" errors escalated 22->33->125/day, with DB commands
blocking 60-102s, causing healthcheck flaps and playback stalls. Config
rsynced from /mnt/docker-data/configs/jellyfin to local /var/lib/jellyfin
on bart; healthy after redeploy. jellyfin:2026060106 v12 image pin
unchanged. Old NFS copy left in place as backup.

OpenVAS fully removed from bart per user (~39GB reclaimed: containers,
volumes incl. PG DB, image). Prometheus probe and Farm Manager link
removed. Compose dir kept for reference; Known Issue #21 marked moot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
jeremy
2026-07-19 18:21:20 -05:00
parent 8a051939ed
commit bfb9b03a26
2 changed files with 9 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+5 -5
View File
@@ -46,9 +46,9 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)
bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13, NFS-lock crashes). **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12.
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12.
├── [REMOVED] Onyx -- fully decommissioned 2026-07-13 (had been down since ~2026-06-08 unnoticed; containers, volumes, images removed)
├── [COMPOSE] OpenVAS (vulnerability scanner) -- Watchtower disabled 2026-07-13 after :latest major pull forced pg_upgrade crash-loop. **2026-07-15: DB wiped, fresh PG15** -- the 13->15 pg_upgrade loop (158 restarts, ~21h) saturated bart's HDD and made Jellyfin hang; upgrade could never finish because the openvas-db volume is mounted AT /data/database, so the image's `mv /data/database ...` swap fails (Device or resource busy). Scan history lost; feeds re-synced. See Known Issue #21.
├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15.
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
@@ -193,7 +193,6 @@ compose/ # Standalone Compose definitions (deployed per-node)
- SABnzbd: :8082
- NZBHydra2: :5076
- Jellyseerr: :5055
- OpenVAS: :9392
**GPU Machine (192.168.86.150, RTX 3090):**
- **NOTE (2026-07-05):** .150 is actually the user's **Windows workstation** (runs the WSL/Claude Code session), not a headless Linux server. It runs **Docker Desktop** (WSL2 backend, RTX 3090 GPU passthrough working) alongside **UniFi OS Server** (Podman). Docker Desktop panics if UniFi/Podman claims `\\.\pipe\docker_engine` first — start Docker Desktop before UniFi OS Server. See memory `docker-desktop-unifi-podman-conflict`.
@@ -304,9 +303,10 @@ All persistent data on NAS at `/mnt/docker-data/`:
13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart; jellyfin config → `/var/lib/jellyfin` on bart 2026-07-19 after 60-100s "database is locked" stalls on jellyfin.db caused playback lockups). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`).
18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router.
19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.
20. **UniFi OS Server probe port**: `unifi-os-server` on .100 serves HTTPS on host port **11443** (container 443), not 8443. Blackbox probe fixed 2026-07-13. solo.hobbs.farm returns 401 (Traefik basicAuth) when healthy — probed via `http_2xx_or_401` module.
21. **OpenVAS pg_upgrade loop saturated bart's HDD (2026-07-15)**: `immauss/openvas:latest` looped on the PG 13→15 auto-upgrade for ~21h (158 restarts) — each cycle re-copied the 21GB DB, pinning sda at ~90% util/40% iowait and making Jellyfin (same disk) hang. The upgrade itself succeeded every time but the final `mv /data/database /data/database_13_old` failed with "Device or resource busy" because the openvas-db volume is mounted **at** `/data/database` (can't rename a mountpoint). Fixed by wiping the DB (`docker volume rm openvas_openvas-db`) and letting it initdb fresh on PG15 — scan history lost, feeds re-synced. **On the next major PG bump this will recur** unless the upgrade is done out-of-band (content-level swap) or the volume is remounted at `/data`. General rule: a service "hanging" on bart usually means another container is eating the single HDD — check `iostat -x`/`pidstat -d` first.
21. **OpenVAS pg_upgrade loop saturated bart's HDD (2026-07-15)**: `immauss/openvas:latest` looped on the PG 13→15 auto-upgrade for ~21h (158 restarts) — each cycle re-copied the 21GB DB, pinning sda at ~90% util/40% iowait and making Jellyfin (same disk) hang. The upgrade itself succeeded every time but the final `mv /data/database /data/database_13_old` failed with "Device or resource busy" because the openvas-db volume is mounted **at** `/data/database` (can't rename a mountpoint). Fixed by wiping the DB (`docker volume rm openvas_openvas-db`) and letting it initdb fresh on PG15 — scan history lost, feeds re-synced. **On the next major PG bump this will recur** unless the upgrade is done out-of-band (content-level swap) or the volume is remounted at `/data`. General rule: a service "hanging" on bart usually means another container is eating the single HDD — check `iostat -x`/`pidstat -d` first. **OpenVAS fully decommissioned 2026-07-19** (moot unless redeployed).
22. **Jellyfin SQLite-on-NFS lockups (fixed 2026-07-19)**: jellyfin.db (WAL mode) on `/mnt/docker-data/configs/jellyfin` hit escalating `SQLite Error 5: 'database is locked'` (22→33→125/day), DB commands blocking 60-102s → healthcheck flaps + playback stalls ("Unexpected end of request content" on /Sessions/Playing/Progress). Config moved to LOCAL `/var/lib/jellyfin` (old copy left on NFS as backup). Same class as Known Issue #16.