docs: OpenVAS pg_upgrade mountpoint loop (Known Issue #21 ) — DB wiped fresh 2026-07-15

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix: repin jellyfin to v12 tag 2026060106 (DB is v12 schema)
2026-07-14 20:03:31 -05:00 · 2026-07-13 21:24:21 -05:00 · 2026-07-13 20:39:14 -05:00 · 2026-07-13 20:24:29 -05:00 · 2026-07-13 20:20:37 -05:00 · 2026-07-13 19:20:10 -05:00
17 changed files with 114 additions and 469 deletions
@@ -23,7 +23,7 @@ This repository manages a **homelab infrastructure** with self-hosted services r
 hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
 ├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
 ├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
-├── [COMPOSE] clip-service (:5200) -- CLIP embeddings (ViT-L-14, CPU)
+├── [REMOVED] clip-service -- decommissioned 2026-07-13 (CLIP embeddings; ~2GB RAM freed, no config references from frigate-notify existed)
 ├── [COMPOSE] Home Assistant (:8123)
 ├── [COMPOSE] ESPHome (:6052)
 ├── [COMPOSE] Mosquitto (:1883, :8883)
@@ -31,35 +31,38 @@ hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, ha
 ├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
 ├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
 ├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
-├── [COMPOSE] AdGuard Home (:3000, :53)
+├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192.
 ├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
 ├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
 ├── [COMPOSE] Farm Agent (:8889)
 ├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
 ├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
 ├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
+├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
+├── [COMPOSE] synapse-admin -- Synapse admin UI
 ├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
 ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
-├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (daily 4am)
+├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (4x daily 4am/10am/4pm/10pm, log ~/frigate-restart.log)
 ├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)

 bart (15GB RAM, i5-4570T, HDD)
-├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr)
-├── [COMPOSE] Onyx (:3005) -- redeployed 2026-05-20, all services :latest, Watchtower disabled on all 10
-├── [COMPOSE] OpenVAS (vulnerability scanner)
+├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13, NFS-lock crashes). **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12.
+├── [REMOVED] Onyx -- fully decommissioned 2026-07-13 (had been down since ~2026-06-08 unnoticed; containers, volumes, images removed)
+├── [COMPOSE] OpenVAS (vulnerability scanner) -- Watchtower disabled 2026-07-13 after :latest major pull forced pg_upgrade crash-loop. **2026-07-15: DB wiped, fresh PG15** -- the 13->15 pg_upgrade loop (158 restarts, ~21h) saturated bart's HDD and made Jellyfin hang; upgrade could never finish because the openvas-db volume is mounted AT /data/database, so the image's `mv /data/database ...` swap fails (Device or resource busy). Scan history lost; feeds re-synced. See Known Issue #21.
 ├── [COMPOSE] Farm Agent (:8889)
 ├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
 ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
 ├── [COMPOSE] Watchtower

 hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
-├── [COMPOSE] Unifi + unifi-db (MongoDB)
-├── [COMPOSE] Gitea + gitea-db (PostgreSQL)
-├── [COMPOSE] Semaphore + semaphore-db (MySQL)
+├── [COMPOSE] UniFi OS Server (container `unifi-os-server`, HTTPS on host :11443)
+├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
+├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
 ├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
-├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy
+├── [COMPOSE] Vaultwarden, MCPO, Trivy
+├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget)
 ├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
-├── [COMPOSE] LifeOS (:8095) -- personal life/goals/habits tracker (PWA). Local mode: Flask+SQLite, no Supabase. LAN-only (no auth in backend). AI companion -> Ollama on .150.
+├── [COMPOSE] CouchDB (:5984) -- Obsidian LiveSync backend for the Second Brain vault. LAN-only. LOCAL disk (NOT NFS).
 ├── [COMPOSE] Farm Agent (:8889)
 ├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
 ├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
@@ -151,14 +154,12 @@ compose/             # Standalone Compose definitions (deployed per-node)
 **hf-pdocker-01 (192.168.86.192):**
 - Frigate: :5000
 - Frigate Alerts (v2): :5199 (web UI + REST API)
- clip-service: :5200 (CLIP embeddings, ViT-L-14, CPU)
 - Home Assistant: :8123
 - ESPHome: :6052
 - Mosquitto: :1883
 - Matrix/Synapse: :8008
 - Well-known (Matrix federation): :8010
 - Guacamole: :8199
- AdGuard Home: :3000 (DNS: :53)
 - Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
 - Grafana: :3001
 - Prometheus: :9095
@@ -175,15 +176,14 @@ compose/             # Standalone Compose definitions (deployed per-node)

 **hf-pdocker-02 (192.168.86.100):**
 - Farm Agent: :8889
- Unifi: :8443
+- UniFi OS Server: :11443 (HTTPS, container 443)
 - Gitea: :3003 (web), :2222 (SSH)
 - Semaphore: :3004
 - Wiki.js: :3002
 - Vaultwarden: :26226
- Actual Budget: :5006
 - MCPO: :8001
 - Trivy: :4954
- LifeOS: :8095 (host) -> :8080 (container). LAN-only, deliberately NOT behind Traefik: the Flask backend is single-user with NO auth. Image `lifeos:local` built on-node from /opt/lifeos (upstream https://github.com/lifeos-app/lifeos, pinned 3009bf6). SQLite at /var/lib/lifeos/data.db on LOCAL disk (NFS breaks SQLite WAL locking). Compose: compose/hf-pdocker-02/lifeos/. Note: host 8080 is taken by unifi-os-server.
+- CouchDB: :5984 -- Obsidian LiveSync backend. DB `secondbrain`; admin creds in Vaultwarden ("CouchDB - Obsidian LiveSync"). LAN-only (no Traefik route). Data on LOCAL disk `/var/lib/couchdb` (NFS breaks CouchDB file locking). **`local.ini` must be mounted READ-WRITE** -- with `:ro` CouchDB exits 1 and logs nothing. Auth required on everything except `/_up` (left open for the blackbox probe). Vault repo: git.hobbs.farm/jeremy/second-brain (private); vault lives at `C:\Users\Jeremy\second-brain` on the Windows box.

 **bart (192.168.86.167):**
 - Farm Agent: :8889
@@ -208,7 +208,7 @@ compose/             # Standalone Compose definitions (deployed per-node)

 Port conflicts already resolved:
 - Port 8000: Freed (MCPO moved to 8001, Portainer moved elsewhere)
- Port 3000: AdGuard Home (Gitea moved to 3003)
+- Port 3000: freed 2026-07-13 (was AdGuard Home, decommissioned; Gitea remains on 3003)
 - Port 3001: Grafana (Semaphore moved to 3004)
 - Port 5000: Frigate uses host networking (registry container stopped)
 - AI Agent ports re-spaced (2026-03-15): openclaw 2026.3.13 uses gateway+2 and gateway+3 for browser/chrome-relay. Agents spaced 5 apart: margaret=18790, spirit=18795
@@ -232,6 +232,7 @@ After ANY infrastructure change (deploy, remove, reconfigure, add service, chang
 2. **MEMORY.md** — Add/update relevant memory entries
 3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus
 4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
+5. **Farm Manager links page** — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)

 Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.

@@ -294,12 +295,18 @@ All persistent data on NAS at `/mnt/docker-data/`:
 4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running
 5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
 6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
-7. **systemd-resolved DNSStubListener**: Must be disabled on any node running AdGuard Home (port 53 conflict). Fix: `/etc/systemd/resolved.conf.d/nostub.conf` with `DNSStubListener=no`, then `systemctl restart systemd-resolved`. Applied on hf-pdocker-01 and hf-pdocker-02.
+7. **systemd-resolved DNSStubListener**: disabled on hf-pdocker-01 and hf-pdocker-02 via `/etc/systemd/resolved.conf.d/nostub.conf` (`DNSStubListener=no`) — originally for AdGuard Home's port 53. AdGuard was decommissioned 2026-07-13; the stub-listener change is harmless and left in place (re-enable only if something needs 127.0.0.53).
 8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`.
 9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
 10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
 11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
-12. **hf-pdocker-01 memory pressure (2026-07-05)**: Node runs hot (~15/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer (~9GB/12GB). Frigate has no semantic-search/face/LPR features enabled.
+12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node.
 13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
 14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
 15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
+16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
+17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`).
+18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router.
+19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.
+20. **UniFi OS Server probe port**: `unifi-os-server` on .100 serves HTTPS on host port **11443** (container 443), not 8443. Blackbox probe fixed 2026-07-13. solo.hobbs.farm returns 401 (Traefik basicAuth) when healthy — probed via `http_2xx_or_401` module.
+21. **OpenVAS pg_upgrade loop saturated bart's HDD (2026-07-15)**: `immauss/openvas:latest` looped on the PG 13→15 auto-upgrade for ~21h (158 restarts) — each cycle re-copied the 21GB DB, pinning sda at ~90% util/40% iowait and making Jellyfin (same disk) hang. The upgrade itself succeeded every time but the final `mv /data/database /data/database_13_old` failed with "Device or resource busy" because the openvas-db volume is mounted **at** `/data/database` (can't rename a mountpoint). Fixed by wiping the DB (`docker volume rm openvas_openvas-db`) and letting it initdb fresh on PG15 — scan history lost, feeds re-synced. **On the next major PG bump this will recur** unless the upgrade is done out-of-band (content-level swap) or the volume is remounted at `/data`. General rule: a service "hanging" on bart usually means another container is eating the single HDD — check `iostat -x`/`pidstat -d` first.
@@ -1,16 +1,27 @@
 services:
  jellyfin:
-    image: jellyfin/jellyfin:latest
+    # PINNED to the 12.0.0 unstable build that migrated the DB to the v12
+    # schema (DropExtraIdsColumn) — :latest is still 10.11.x and CANNOT read
+    # this DB ("no such column: b.ExtraIds"). Do not unpin until official
+    # stable >= 12 ships. (2026-07-14: a :latest deploy downgraded it and
+    # broke Resume/home sections; repinned.)
+    image: jellyfin/jellyfin:2026060106
    container_name: jellyfin
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
+      - LIBVA_DRIVER_NAME=i965   # Haswell HD 4600 uses legacy i965 (iHD is Broadwell+)
    volumes:
      - /mnt/docker-data/configs/jellyfin:/config
      - /mnt/nas-media:/media
    ports:
      - "8096:8096"
+    devices:
+      - /dev/dri:/dev/dri        # Intel iGPU for VAAPI hardware transcoding
+    group_add:
+      - "993"   # render
+      - "44"    # video
    mem_limit: 2G
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:8096/health || exit 1"]
@@ -33,7 +44,9 @@ services:
      - PGID=1000
      - TZ=America/Chicago
    volumes:
-      - /mnt/docker-data/configs/sonarr:/config
+      # LOCAL disk, not NFS: SQLite-on-NFS locking crashed Sonarr's DB migration
+      # after the 2026-07-12 power event ("waiting for user intervention")
+      - /var/lib/sonarr:/config
      - /mnt/nas-media:/media
    ports:
      - "8989:8989"
@@ -111,10 +124,12 @@ services:
      - PGID=1000
      - TZ=America/Chicago
    volumes:
-      - /mnt/docker-data/configs/nzbhydra:/config
+      # LOCAL disk, not NFS: stale NFS lock on the H2 database after the
+      # 2026-07-12 power event kept hydra unhealthy in a DB-retry loop
+      - /var/lib/nzbhydra:/config
    ports:
      - "5076:5076"
-    mem_limit: 768M
+    mem_limit: 1G
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:5076/ || exit 1"]
      interval: 30s
@@ -1,231 +0,0 @@
-services:
-  api_server:
-    image: onyxdotapp/onyx-backend:latest
-    command: >
-      /bin/sh -c "alembic upgrade head &&
-      echo Starting Onyx API Server &&
-      uvicorn onyx.main:app --host 0.0.0.0 --port 8080"
-    depends_on:
-      relational_db:
-        condition: service_healthy
-      index:
-        condition: service_healthy
-      cache:
-        condition: service_healthy
-    restart: "no"
-    mem_limit: 2G
-    memswap_limit: 2G
-    env_file:
-      - .env
-    environment:
-      - AUTH_TYPE=basic
-      - DISABLE_TELEMETRY=true
-      - POSTGRES_HOST=relational_db
-      - VESPA_HOST=index
-      - REDIS_HOST=cache
-      - MODEL_SERVER_HOST=inference_model_server
-      - INDEXING_MODEL_SERVER_HOST=indexing_model_server
-      - OPENSEARCH_HOST=opensearch
-      - ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false
-      - FILE_STORE_BACKEND=postgres
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - onyx_file_connector:/home/file_connector_storage
-    networks:
-      - onyx
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  background:
-    image: onyxdotapp/onyx-backend:latest
-    command: >
-      /bin/sh -c "/usr/bin/supervisord -c /etc/supervisor/conf.d/supervisord.conf"
-    depends_on:
-      relational_db:
-        condition: service_healthy
-      index:
-        condition: service_healthy
-      cache:
-        condition: service_healthy
-    restart: "no"
-    mem_limit: 3G
-    memswap_limit: 3G
-    env_file:
-      - .env
-    environment:
-      - AUTH_TYPE=basic
-      - DISABLE_TELEMETRY=true
-      - POSTGRES_HOST=relational_db
-      - VESPA_HOST=index
-      - REDIS_HOST=cache
-      - MODEL_SERVER_HOST=inference_model_server
-      - INDEXING_MODEL_SERVER_HOST=indexing_model_server
-      - OPENSEARCH_HOST=opensearch
-      - ENABLE_OPENSEARCH_INDEXING_FOR_ONYX=false
-      - FILE_STORE_BACKEND=postgres
-    extra_hosts:
-      - "host.docker.internal:host-gateway"
-    volumes:
-      - onyx_file_connector:/home/file_connector_storage
-    networks:
-      - onyx
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  web_server:
-    image: onyxdotapp/onyx-web-server:latest
-    depends_on:
-      - api_server
-    restart: "no"
-    mem_limit: 512M
-    memswap_limit: 512M
-    environment:
-      - INTERNAL_URL=http://api_server:8080
-      - WEB_DOMAIN=https://onyx.hobbs.farm
-      - NEXT_PUBLIC_DISABLE_STREAMING=false
-      - AUTH_TYPE=basic
-    networks:
-      - onyx
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  inference_model_server:
-    image: onyxdotapp/onyx-model-server:latest
-    command: >
-      /bin/sh -c "exec uvicorn model_server.main:app --host 0.0.0.0 --port 9000"
-    restart: "no"
-    mem_limit: 2G
-    memswap_limit: 2G
-    environment:
-      - MIN_THREADS_ML_MODELS=1
-      - LOG_LEVEL=info
-    volumes:
-      - onyx_model_cache_huggingface:/root/.cache/huggingface/
-    networks:
-      - onyx
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  indexing_model_server:
-    image: onyxdotapp/onyx-model-server:latest
-    command: >
-      /bin/sh -c "exec uvicorn model_server.main:app --host 0.0.0.0 --port 9000"
-    restart: "no"
-    mem_limit: 2G
-    memswap_limit: 2G
-    environment:
-      - MIN_THREADS_ML_MODELS=1
-      - INDEXING_ONLY=true
-      - LOG_LEVEL=info
-    volumes:
-      - onyx_model_cache_huggingface:/root/.cache/huggingface/
-    networks:
-      - onyx
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  relational_db:
-    image: postgres:15.2-alpine
-    restart: "no"
-    mem_limit: 512M
-    memswap_limit: 512M
-    environment:
-      - POSTGRES_USER=postgres
-      - POSTGRES_PASSWORD=onyx_postgres_pw
-    volumes:
-      - onyx_db:/var/lib/postgresql/data
-    networks:
-      - onyx
-    healthcheck:
-      test: ["CMD-SHELL", "pg_isready -U postgres"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  index:
-    image: vespaengine/vespa:8.609.39
-    restart: "no"
-    mem_limit: 4G
-    memswap_limit: 4G
-    volumes:
-      - onyx_vespa:/opt/vespa/var
-    networks:
-      - onyx
-    healthcheck:
-      test: ["CMD-SHELL", "curl -sf http://localhost:19071/state/v1/health || exit 1"]
-      interval: 30s
-      timeout: 10s
-      retries: 10
-      start_period: 120s
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  opensearch:
-    image: opensearchproject/opensearch:2.11.1
-    restart: "no"
-    mem_limit: 2G
-    memswap_limit: 2G
-    environment:
-      - discovery.type=single-node
-      - plugins.security.disabled=true
-      - OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g
-      - DISABLE_INSTALL_DEMO_CONFIG=true
-    volumes:
-      - onyx_opensearch:/usr/share/opensearch/data
-    networks:
-      - onyx
-    healthcheck:
-      test: ["CMD-SHELL", "curl -sf http://localhost:9200/_cluster/health || exit 1"]
-      interval: 30s
-      timeout: 10s
-      retries: 10
-      start_period: 120s
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  cache:
-    image: redis:7.4-alpine
-    restart: "no"
-    mem_limit: 256M
-    memswap_limit: 256M
-    command: redis-server --maxmemory 128mb --maxmemory-policy allkeys-lru
-    networks:
-      - onyx
-    healthcheck:
-      test: ["CMD", "redis-cli", "ping"]
-      interval: 10s
-      timeout: 5s
-      retries: 5
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-  nginx:
-    image: nginx:1.25.5-alpine
-    restart: "no"
-    mem_limit: 128M
-    memswap_limit: 128M
-    depends_on:
-      - api_server
-      - web_server
-    ports:
-      - "3005:80"
-    volumes:
-      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
-    networks:
-      - onyx
-    labels:
-      com.centurylinklabs.watchtower.enable: "false"
-
-volumes:
-  onyx_file_connector:
-  onyx_model_cache_huggingface:
-  onyx_db:
-  onyx_vespa:
-  onyx_opensearch:
-
-networks:
-  onyx:
-    driver: bridge
@@ -1,36 +0,0 @@
-server {
-    listen 80;
-    server_name _;
-
-    # Use Docker's internal DNS with a short TTL so container IP changes
-    # (e.g. after docker compose up -d for a single service) are picked up
-    # without requiring an nginx reload.
-    resolver 127.0.0.11 valid=10s;
-
-    client_max_body_size 100M;
-
-    location /api/ {
-        set $api_upstream http://api_server:8080;
-        rewrite ^/api/(.*)$ /$1 break;
-        proxy_pass $api_upstream;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_read_timeout 300s;
-        proxy_connect_timeout 75s;
-        proxy_buffering off;
-        proxy_cache off;
-    }
-
-    location / {
-        set $web_upstream http://web_server:3000;
-        proxy_pass $web_upstream;
-        proxy_set_header Host $host;
-        proxy_set_header X-Real-IP $remote_addr;
-        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
-        proxy_set_header X-Forwarded-Proto $scheme;
-        proxy_read_timeout 300s;
-        proxy_connect_timeout 75s;
-    }
-}
@@ -3,6 +3,10 @@ services:
    image: immauss/openvas:latest
    container_name: openvas
    restart: unless-stopped
+    labels:
+      # Watchtower pulled a new major that forced a pg_upgrade crash-loop
+      # (212 restarts, 2026-07-13); update OpenVAS deliberately instead
+      - "com.centurylinklabs.watchtower.enable=false"
    environment:
      - PASSWORD=${OPENVAS_PASSWORD}
      - USERNAME=${OPENVAS_USERNAME}
@@ -1,26 +0,0 @@
-services:
-  adguardhome:
-    image: adguard/adguardhome:latest
-    container_name: adguardhome
-    restart: unless-stopped
-    volumes:
-      - /mnt/docker-data/configs/adguardhome/work:/opt/adguardhome/work
-      - /mnt/docker-data/configs/adguardhome/conf:/opt/adguardhome/conf
-    ports:
-      - "53:53/tcp"
-      - "53:53/udp"
-      - "3000:3000/tcp"
-      - "853:853/tcp"
-    mem_limit: 256M
-    memswap_limit: 256M
-    healthcheck:
-      test: ["CMD-SHELL", "wget -q --spider http://localhost:3000/ || exit 1"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 30s
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "50m"
-        max-file: "3"
@@ -1,19 +0,0 @@
-services:
-  clip-service:
-    image: clip-service:latest
-    container_name: clip-service
-    restart: unless-stopped
-    network_mode: host
-    mem_limit: 3G
-    memswap_limit: 3G
-    healthcheck:
-      test: ["CMD-SHELL", "curl -sf http://localhost:5200/health || exit 1"]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 180s
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "50m"
-        max-file: "3"
@@ -1,22 +0,0 @@
-services:
-  actual-budget:
-    image: actualbudget/actual-server:latest
-    container_name: actual-budget
-    restart: unless-stopped
-    ports:
-      - "5006:5006"
-    volumes:
-      - /mnt/docker-data/configs/actual-budget:/data
-    mem_limit: 256M
-    memswap_limit: 256M
-    healthcheck:
-      test: ["CMD-SHELL", "node -e \"fetch('http://localhost:5006/').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))\""]
-      interval: 30s
-      timeout: 10s
-      retries: 3
-      start_period: 60s
-    logging:
-      driver: "json-file"
-      options:
-        max-size: "50m"
-        max-file: "3"
@@ -0,0 +1,2 @@
+COUCHDB_USER=obsidian
+COUCHDB_PASSWORD=REPLACE_AT_DEPLOY
@@ -0,0 +1,27 @@
+services:
+  couchdb:
+    image: couchdb:3.4
+    container_name: couchdb
+    restart: unless-stopped
+    ports:
+      - "5984:5984"
+    environment:
+      - COUCHDB_USER=${COUCHDB_USER}
+      - COUCHDB_PASSWORD=${COUCHDB_PASSWORD}
+      - TZ=America/Chicago
+    volumes:
+      # LOCAL disk, not NFS: CouchDB's file locking breaks over NFSv3, the same
+      # way it did for guacamole-db (MySQL) and semaphore-db.
+      - /var/lib/couchdb/data:/opt/couchdb/data
+      # NOT :ro -- CouchDB 3.x writes runtime config back into the last-loaded ini.
+      # With a read-only mount it exits 1 immediately and logs NOTHING at all.
+      - /var/lib/couchdb/config/local.ini:/opt/couchdb/etc/local.d/local.ini
+    labels:
+      - "com.centurylinklabs.watchtower.enable=false"
+    mem_limit: 1g
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://localhost:5984/_up"]
+      interval: 60s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
@@ -0,0 +1,30 @@
+; CouchDB config for Obsidian LiveSync.
+; Obsidian talks to CouchDB from an app:// origin, so CORS must allow it explicitly --
+; a wildcard origin is not permitted alongside credentials.
+
+[couchdb]
+single_node = true
+max_document_size = 50000000
+
+[chttpd]
+require_valid_user = true
+; ...but leave /_up open, so the Prometheus blackbox probe can health-check it
+; without credentials. Everything else still returns 401 to anonymous callers.
+require_valid_user_except_for_up = true
+max_http_request_size = 4294967296
+bind_address = 0.0.0.0
+
+[chttpd_auth]
+require_valid_user = true
+authentication_redirect = /_utils/session.html
+
+[httpd]
+WWW-Authenticate = Basic realm="couchdb"
+enable_cors = true
+
+[cors]
+credentials = true
+; Obsidian desktop is app://obsidian.md; mobile uses capacitor://localhost.
+origins = app://obsidian.md,capacitor://localhost,http://localhost
+methods = GET,PUT,POST,HEAD,DELETE
+headers = accept,authorization,content-type,origin,referer,x-csrf-token
@@ -19,6 +19,9 @@ services:
      - GITEA__server__SSH_PORT=2222
    volumes:
      - /mnt/docker-data/configs/gitea:/data
+      # LOCAL disk for LevelDB queues: stale NFS lock after 2026-07-12 power event
+      # caused a 28h crash loop (7,972 restarts)
+      - /var/lib/gitea-queues:/data/gitea/queues
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    depends_on:
@@ -1,51 +0,0 @@
-# LifeOS — https://github.com/lifeos-app/lifeos
-#
-# Two stages: Node builds the Vite/React bundle, Python serves that bundle plus
-# the Flask API from a single process. Build context is the upstream source tree
-# (cloned to /opt/lifeos on the node); this file is copied in alongside it.
-
-# ─── Stage 1: build the frontend ──────────────────────────────────────
-FROM node:22-bookworm-slim AS frontend
-WORKDIR /build
-
-COPY package.json package-lock.json ./
-# --ignore-scripts skips the electron-builder postinstall, which pulls a desktop
-# toolchain we never use and fails without an X stack.
-RUN npm ci --ignore-scripts
-
-COPY . .
-
-# Routes the data layer (src/lib/data-access.ts) to the local Flask API instead
-# of Supabase Cloud. VITE_API_BASE_URL is deliberately left unset: local-api.ts
-# then falls back to a relative /api, so the bundle works on any host and port.
-ENV VITE_USE_LOCAL_API=true
-ENV NODE_OPTIONS=--max-old-space-size=3072
-RUN npm run build
-
-# ─── Stage 2: Flask API + built frontend ──────────────────────────────
-FROM python:3.12-slim
-WORKDIR /app
-
-COPY backend/requirements.txt ./
-RUN pip install --no-cache-dir -r requirements.txt gunicorn
-
-COPY backend/ /app/
-COPY serve_container.py /app/
-COPY --from=frontend /build/dist /app/dist
-
-# Upstream hardcodes the author's own model name in the chat proxy, and a 30s
-# timeout that a cold Ollama model load (~46s for a 9.6GB model) blows through.
-# Make both env vars. Fails the build if upstream renames these out from under us.
-RUN sed -i 's/"model": "sentient-teddy"/"model": os.environ.get("AI_MODEL", "sentient-teddy")/' app.py \
-    && sed -i 's/timeout=30$/timeout=int(os.environ.get("AI_TIMEOUT", "120"))/' app.py \
-    && grep -q 'os.environ.get("AI_MODEL"' app.py \
-    && grep -q 'os.environ.get("AI_TIMEOUT"' app.py
-
-# app.py hardcodes its SQLite path to ~/.lifeos/data.db, so HOME is what pins
-# the database into the mounted volume.
-ENV HOME=/data
-RUN mkdir -p /data/.lifeos
-
-EXPOSE 8080
-CMD ["gunicorn", "--bind", "0.0.0.0:8080", "--workers", "2", "--threads", "4", \
-     "--timeout", "60", "--access-logfile", "-", "serve_container:app"]
@@ -1,32 +0,0 @@
-services:
-  lifeos:
-    build:
-      context: /opt/lifeos
-      dockerfile: Dockerfile
-    image: lifeos:local
-    container_name: lifeos
-    restart: unless-stopped
-    ports:
-      # 8080 on this node is taken by unifi-os-server.
-      - "8095:8080"
-    environment:
-      - TZ=America/Chicago
-      # AI companion -> Ollama on the GPU machine (OpenAI-compatible endpoint).
-      - AI_BRIDGE_URL=http://192.168.86.150:11434
-      - AI_MODEL=gemma4:latest
-      # Ollama needs ~46s to cold-load gemma4 (0.7s once warm); upstream's
-      # hardcoded 30s timeout would fail every first message.
-      - AI_TIMEOUT=120
-    volumes:
-      # SQLite (WAL) stays on local disk: NFS v3 locking breaks it, same reason
-      # guacamole-db lives on /var/lib.
-      - /var/lib/lifeos:/data/.lifeos
-    labels:
-      - "com.centurylinklabs.watchtower.enable=false"
-    mem_limit: 1g
-    healthcheck:
-      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8080/api/health-check').read()"]
-      interval: 60s
-      timeout: 10s
-      retries: 3
-      start_period: 30s
@@ -1,28 +0,0 @@
-"""Container entrypoint: serve the Flask API and the built SPA from one process.
-
-Upstream's backend/serve.py does this too, but it also shells out to `npm run
-build` and tries to launch a browser, neither of which works in a container.
-"""
-
-from pathlib import Path
-
-from flask import send_file, send_from_directory
-
-from app import app  # importing app runs init_db(), creating the SQLite schema
-
-DIST = Path(__file__).parent / "dist"
-
-
-@app.route("/")
-def _index():
-    return send_file(DIST / "index.html")
-
-
-@app.route("/<path:path>")
-def _static(path):
-    asset = DIST / path
-    if asset.is_file():
-        return send_from_directory(DIST, path)
-    if path.startswith("api/"):
-        return {"error": "Not found"}, 404
-    return send_file(DIST / "index.html")  # SPA client-side routing
@@ -53,7 +53,9 @@ services:
      - MYSQL_USER=semaphore
      - MYSQL_PASSWORD=${SEMAPHORE_DB_PASSWORD}
    volumes:
-      - /mnt/docker-data/databases/semaphore-db:/var/lib/mysql
+      # LOCAL disk, not NFS: MySQL on NFS suffers stale-lock and InnoDB corruption
+      # (2026-07-11 corruption, 2026-07-13 stale ibdata1 lock after power event)
+      - /var/lib/semaphore-db:/var/lib/mysql
    networks:
      - semaphore-internal
    labels:
Author	SHA1	Message	Date
jeremy	8a051939ed	docs: OpenVAS pg_upgrade mountpoint loop (Known Issue #21 ) — DB wiped fresh 2026-07-15 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-14 20:03:31 -05:00
jeremy	66c65d1b4e	fix: repin jellyfin to v12 tag 2026060106 (DB is v12 schema) The 2026-07-13 media redeploy overwrote a node-side pin with the repo's :latest (10.11.x), downgrading jellyfin under its v12-migrated DB and breaking Resume/home queries ('no such column: b.ExtraIds'). Repinned; do not unpin until official stable >= 12. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-13 21:24:21 -05:00
jeremy	45573cf9fc	fix: nzbhydra2 stale NFS H2 lock -> local /var/lib/nzbhydra; add Farm Manager links page to post-change checklist nzbhydra2 was the 4th victim of the 2026-07-12 power-event stale NFS locks (H2 'file is locked' retry loop; the earlier mem bump was a red herring). Farm Manager links updated (AdGuard/Actual Budget/Portainer removed, UniFi -> :11443) and image rebuilt; links maintenance is now checklist item 5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-13 20:39:14 -05:00
jeremy	d35af1ddad	chore: decommission clip-service (.192) and Actual Budget (.100) clip-service: container+volumes+compose removed, ~2GB RAM freed; frigate-notify config never referenced it. Actual Budget: container+compose removed, data kept at /mnt/docker-data/configs/actual-budget. Prometheus probes for both removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-13 20:24:29 -05:00
jeremy	ed5f330d2c	chore: decommission AdGuard Home (dead since 2026-05-17, LAN uses router DNS) Container, data, compose dir, and Prometheus probe removed from .192. Ports 53 and 3000 freed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-13 20:20:37 -05:00
jeremy	5d7c8ea030	fix: recover from 2026-07-12 power-event fallout + efficiency cleanup - gitea: LevelDB queues -> local /var/lib/gitea-queues (stale NFS lock, 7,972-restart loop) - semaphore-db: data -> local /var/lib/semaphore-db (stale ibdata1 NFS lock, 975 restarts) - sonarr: config -> local /var/lib/sonarr (SQLite-on-NFS migration crash) - nzbhydra2: mem_limit 768M -> 1G (was pinned at limit, unhealthy) - jellyfin: deploy pending VAAPI device passthrough (i965/renderD128) - openvas: disable Watchtower (latest major pull forced pg_upgrade crash-loop) - onyx: fully decommissioned (down since ~June 8 unnoticed) - remove duplicate monitoring/traefik/guacamole stacks from .100, failover guacamole from bart - synapse: raise rc_message (alertmanager receiver was 429-limited; alerts silently dropped for weeks) - prometheus: UniFi probe 8443 -> 11443, solo.hobbs.farm via 401-tolerant module, drop onyx target - frigate restart cron: unwritable /var/log path meant it never ran; now 4x daily to ~glyptic - CLAUDE.md: document all of the above + AdGuard broken/unconfigured since 2026-05-17 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-13 19:20:10 -05:00
jeremy	81a7464dd8	docs: record CouchDB/Obsidian LiveSync in CLAUDE.md Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-11 12:04:02 -05:00
jeremy	0e4b2f7d1a	feat: deploy CouchDB on hf-pdocker-02 for Obsidian LiveSync (:5984) Backs the Obsidian 'Second Brain' vault (private repo: git.hobbs.farm/jeremy/ second-brain) with self-hosted multi-device sync. LAN-only for now; no public Traefik route yet. - couchdb:3.4 pinned, Watchtower disabled, mem_limit 1g. - Data on LOCAL disk (/var/lib/couchdb) -- CouchDB file locking breaks over NFSv3, same class as guacamole-db and semaphore-db. - local.ini mounted READ-WRITE, deliberately: CouchDB 3.x writes runtime config back into the last-loaded ini, and with :ro it exits 1 and logs NOTHING -- a silent crash-loop that is very hard to diagnose. - CORS allows app://obsidian.md and capacitor://localhost (LiveSync desktop + mobile origins); a wildcard origin is not allowed alongside credentials. - require_valid_user everywhere EXCEPT /_up, so the blackbox probe can health check it without credentials. Anonymous callers get 401 on all data. - Admin credentials in Vaultwarden ('CouchDB - Obsidian LiveSync'); .env is gitignored, .env.example carries a placeholder. Prometheus probe added (probe_success=1). Vault database: 'secondbrain'. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-11 12:03:15 -05:00
jeremy	f0e2918bd1	revert: remove LifeOS deployment Evaluated and torn down at the user's request. Container, image, source (/opt/lifeos), SQLite data (/var/lib/lifeos) and compose dir removed from hf-pdocker-02; Prometheus probe removed and both instances reloaded; port 8095 released. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-11 11:43:14 -05:00
jeremy	f48c7a40cd	fix: make LifeOS actually render in local mode The container served a blank page. Four upstream assumptions broke a root-served, Supabase-less deployment; all are patched at build time with grep assertions so a silent upstream rename fails the build instead of shipping a blank page again. - React Router was pinned to basename '/app' (upstream is hosted under an /app path). At '/' it matched no route and rendered nothing -- silently, with only a console warning. This was the blank page. - supabase.ts calls createClient() at module scope, which throws 'supabaseUrl is required' on an empty string and killed the bundle before React mounted. Dummy values fix it; nothing reads Supabase in local mode. - initAuth() only booted the local user for Tauri/Electron, so a browser build fell through to a Supabase login screen the backend cannot serve (it has no /api/auth/login or /api/auth/signup -- it is single-user). - The local API adapter has no realtime layer, so supabase.channel() was undefined and every subscribing component tripped the ErrorBoundary ('Something went wrong'). Added no-op stubs; realtime is meaningless on a single-user instance. Verified in a browser: dashboard and routed pages render, 0 fatal errors. Known gap: the Flask backend implements only a subset of the cloud schema, so quests, messages, partnerships, grocery_lists, recurring_transactions, junction_traditions and habit_suggestions 404. Non-fatal; those features degrade. Core (tasks/habits/goals/health/finance/journal/XP/AI chat) works. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-11 11:32:14 -05:00