From d35af1ddadd95d59e2dd2001357b238d914fb0ba Mon Sep 17 00:00:00 2001 From: jeremy Date: Mon, 13 Jul 2026 20:24:29 -0500 Subject: [PATCH] chore: decommission clip-service (.192) and Actual Budget (.100) clip-service: container+volumes+compose removed, ~2GB RAM freed; frigate-notify config never referenced it. Actual Budget: container+compose removed, data kept at /mnt/docker-data/configs/actual-budget. Prometheus probes for both removed. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 9 ++++---- .../clip-service/docker-compose.yml | 19 ---------------- .../actual-budget/docker-compose.yml | 22 ------------------- 3 files changed, 4 insertions(+), 46 deletions(-) delete mode 100644 compose/hf-pdocker-01/clip-service/docker-compose.yml delete mode 100644 compose/hf-pdocker-02/actual-budget/docker-compose.yml diff --git a/CLAUDE.md b/CLAUDE.md index 080b60e..fcc432d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -23,7 +23,7 @@ This repository manages a **homelab infrastructure** with self-hosted services r hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired) ├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active ├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications -├── [COMPOSE] clip-service (:5200) -- CLIP embeddings (ViT-L-14, CPU) +├── [REMOVED] clip-service -- decommissioned 2026-07-13 (CLIP embeddings; ~2GB RAM freed, no config references from frigate-notify existed) ├── [COMPOSE] Home Assistant (:8123) ├── [COMPOSE] ESPHome (:6052) ├── [COMPOSE] Mosquitto (:1883, :8883) @@ -59,7 +59,8 @@ hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD) ├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop) ├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13) ├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL) -├── [COMPOSE] Vaultwarden, Actual Budget, MCPO, Trivy +├── [COMPOSE] Vaultwarden, MCPO, Trivy +├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget) ├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation ├── [COMPOSE] CouchDB (:5984) -- Obsidian LiveSync backend for the Second Brain vault. LAN-only. LOCAL disk (NOT NFS). ├── [COMPOSE] Farm Agent (:8889) @@ -153,7 +154,6 @@ compose/ # Standalone Compose definitions (deployed per-node) **hf-pdocker-01 (192.168.86.192):** - Frigate: :5000 - Frigate Alerts (v2): :5199 (web UI + REST API) -- clip-service: :5200 (CLIP embeddings, ViT-L-14, CPU) - Home Assistant: :8123 - ESPHome: :6052 - Mosquitto: :1883 @@ -181,7 +181,6 @@ compose/ # Standalone Compose definitions (deployed per-node) - Semaphore: :3004 - Wiki.js: :3002 - Vaultwarden: :26226 -- Actual Budget: :5006 - MCPO: :8001 - Trivy: :4954 - CouchDB: :5984 -- Obsidian LiveSync backend. DB `secondbrain`; admin creds in Vaultwarden ("CouchDB - Obsidian LiveSync"). LAN-only (no Traefik route). Data on LOCAL disk `/var/lib/couchdb` (NFS breaks CouchDB file locking). **`local.ini` must be mounted READ-WRITE** -- with `:ro` CouchDB exits 1 and logs nothing. Auth required on everything except `/_up` (left open for the blackbox probe). Vault repo: git.hobbs.farm/jeremy/second-brain (private); vault lives at `C:\Users\Jeremy\second-brain` on the Windows box. @@ -300,7 +299,7 @@ All persistent data on NAS at `/mnt/docker-data/`: 9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset. 10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors. 11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400. -12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by clip-service + frigate-notify v2. **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node. +12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) fully removed (replaced by FRIDAY Voice on .150). Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node. 13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`. 14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`. 15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11). diff --git a/compose/hf-pdocker-01/clip-service/docker-compose.yml b/compose/hf-pdocker-01/clip-service/docker-compose.yml deleted file mode 100644 index 3aff0d4..0000000 --- a/compose/hf-pdocker-01/clip-service/docker-compose.yml +++ /dev/null @@ -1,19 +0,0 @@ -services: - clip-service: - image: clip-service:latest - container_name: clip-service - restart: unless-stopped - network_mode: host - mem_limit: 3G - memswap_limit: 3G - healthcheck: - test: ["CMD-SHELL", "curl -sf http://localhost:5200/health || exit 1"] - interval: 30s - timeout: 10s - retries: 3 - start_period: 180s - logging: - driver: "json-file" - options: - max-size: "50m" - max-file: "3" diff --git a/compose/hf-pdocker-02/actual-budget/docker-compose.yml b/compose/hf-pdocker-02/actual-budget/docker-compose.yml deleted file mode 100644 index 35505e3..0000000 --- a/compose/hf-pdocker-02/actual-budget/docker-compose.yml +++ /dev/null @@ -1,22 +0,0 @@ -services: - actual-budget: - image: actualbudget/actual-server:latest - container_name: actual-budget - restart: unless-stopped - ports: - - "5006:5006" - volumes: - - /mnt/docker-data/configs/actual-budget:/data - mem_limit: 256M - memswap_limit: 256M - healthcheck: - test: ["CMD-SHELL", "node -e \"fetch('http://localhost:5006/').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))\""] - interval: 30s - timeout: 10s - retries: 3 - start_period: 60s - logging: - driver: "json-file" - options: - max-size: "50m" - max-file: "3"