hobbs-iac/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

This repository manages a **homelab infrastructure** with self-hosted services running across 3 nodes using **standalone Docker Compose** on all nodes. (Docker Swarm was fully dismantled on 2026-04-01.)

### Cluster Architecture

**Nodes:**
- **hf-pdocker-01** (192.168.86.192) - Core infrastructure + Frigate NVR - **RESTORED (2026-03-08)**
- **hf-pdocker-02** (192.168.86.100) - Dev tools, utilities
- **bart** (192.168.86.167) - Media stack, OpenVAS

**Synology NAS** (192.168.86.30) - Centralized NFS storage for all service data

**Access:** Username `glyptic`, password `&Rip8a115` (has sudo access on all nodes)

### Service Distribution

```
hf-pdocker-01 (23GB RAM, i7-7700T, NVMe, HD 630 GPU) -- RESTORED (2026-03-08, hardware repaired)
├── [COMPOSE] Frigate NVR (:5000) -- Coral USB TPU (edgetpu), ~11ms inference, 13 cameras active
├── [COMPOSE] Frigate-Notify v2 (:5199) -- MQTT->AI->Matrix notifications
├── [REMOVED] clip-service -- decommissioned 2026-07-13 (CLIP embeddings; ~2GB RAM freed, no config references from frigate-notify existed)
├── [COMPOSE] Home Assistant (:8123)
├── [COMPOSE] ESPHome (:6052)
├── [COMPOSE] Mosquitto (:1883, :8883)
├── [COMPOSE] Synapse/Matrix (:8008) + wellknown (:8010) + synapse-db (**postgres:16-alpine, LOCAL /var/lib/synapse-db, C collation**) -- Synapse data on LOCAL /var/lib/synapse. Migrated SQLite->PostgreSQL 2026-08-20 (required by syn2mas). **Auth is delegated to MAS** -- Synapse no longer serves /login.
├── [COMPOSE] **MAS / Matrix Authentication Service (:8020 web, :8021 health) + mas-db (postgres:16-alpine, LOCAL /var/lib/mas-db)** -- OAuth2/OIDC provider for Matrix. Public: auth.hobbs.farm. Deployed 2026-08-20 to enable QR sign-in (MSC4108). **Every Matrix login now depends on this service.**
├── [COMPOSE] Guacamole (:8199) + guacd + guacamole-db (mysql, LOCAL /var/lib/guacamole-db)
├── [COMPOSE] AI Agents (margaret:18790, spirit:18795 [infra ops])
├── [COMPOSE] Traefik (reverse proxy, ports 80/443/8090)
├── [REMOVED] AdGuard Home -- decommissioned 2026-07-13 (conf dir had been empty since 2026-05-17, DNS engine never ran, LAN uses router DNS). Ports 53/3000 freed on .192.
├── [COMPOSE] Monitoring (Grafana :3001, Prometheus :9095, Alertmanager :9093, InfluxDB :8086, Blackbox :9115, matrix-receiver :9094)
├── [COMPOSE] Farm Manager API (:8888) -- dashboard + API server
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Rongorongo decipherment engine -- 2 containers (rongorongo-web :8181, rongorongo-api). Server-synced state at /mnt/docker-data/configs/rongorongo/data. Traefik basicAuth on rongorongo.hobbs.farm.
├── [COMPOSE] LinkedStorm (:8300 pipeline, :8310 api, cron no-port) -- whole-home audio assistant. VAD+Whisper STT+speaker-ID. Deployed 2026-05-28. Public: linkedstorm.hobbs.farm -> :8310.
├── [REMOVED] VoiceCoach -- moved to GPU machine (192.168.86.150) 2026-04-04
├── [COMPOSE] litellm (:4000?) + litellm-db -- LLM proxy (see Known Issue #15)
├── [COMPOSE] synapse-admin -- Synapse admin UI
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [CRON] sync-frigate-to-nas (daily 2am), cleanup-frigate-media (weekly Sun 3am), restart-frigate (4x daily 4am/10am/4pm/10pm, log ~/frigate-restart.log)
├── 5TB USB disk at /mnt/frigate-recordings (Frigate recordings storage)

bart (15GB RAM, i5-4570T, HDD)
├── [COMPOSE] Media Stack (Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr) -- Sonarr config on LOCAL /var/lib/sonarr, NZBHydra config on LOCAL /var/lib/nzbhydra (2026-07-13), **Jellyfin config on LOCAL /var/lib/jellyfin (2026-07-19)** -- all moved off NFS after SQLite/lock crashes. **Jellyfin PINNED to jellyfin/jellyfin:2026060106 (v12.0.0 unstable)** -- its DB is on the v12 schema (ExtraIds column dropped); :latest is still 10.11.x and breaks on it. Do not unpin until official stable >= 12.
├── [REMOVED] Onyx -- fully decommissioned 2026-07-13 (had been down since ~2026-06-08 unnoticed; containers, volumes, images removed)
├── [REMOVED] OpenVAS -- decommissioned 2026-07-19 per user (containers, volumes incl. PG DB, image removed; ~39GB freed with other cleanup). Compose dir kept for reference. History: pg_upgrade crash-loop (Known Issue #21), DB wiped fresh 2026-07-15.
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Watchtower

hf-pdocker-02 (7.6GB RAM, i7-6700T, HDD)
├── [COMPOSE] UniFi OS Server (container `unifi-os-server`, HTTPS on host :11443)
├── [COMPOSE] Gitea + gitea-db (PostgreSQL) -- LevelDB queues on LOCAL /var/lib/gitea-queues (2026-07-13, stale NFS lock caused 7,972-restart loop)
├── [COMPOSE] Semaphore + semaphore-db (MySQL, LOCAL /var/lib/semaphore-db since 2026-07-13)
├── [COMPOSE] Wiki.js + wikijs-db (PostgreSQL)
├── [COMPOSE] Vaultwarden, MCPO, Trivy
├── [REMOVED] Actual Budget -- decommissioned 2026-07-13 (data kept at /mnt/docker-data/configs/actual-budget)
├── [COMPOSE] n8n + n8n-db (PostgreSQL) -- workflow automation
├── [COMPOSE] CouchDB (:5984) -- Obsidian LiveSync backend for the Second Brain vault. LAN-only. LOCAL disk (NOT NFS).
├── [COMPOSE] Farm Agent (:8889)
├── [COMPOSE] Node-exporter (:9100) -- standalone with pid:host
├── [COMPOSE] Telegraf -- metrics collection to InfluxDB
├── [COMPOSE] Watchtower

```

## Working with This Repository

### Environment Context

You are working in a **WSL 2 environment** on Windows. Docker is NOT directly available in WSL - all Docker operations must be performed by connecting to the remote nodes via SSH.

**Critical:** Docker is not available locally in WSL. All Docker operations must be performed by connecting to the remote nodes via SSH.

### Storage Architecture

All service data lives on the **Synology NAS** via NFS, mounted at `/mnt/docker-data` on all 3 nodes.

```
/mnt/docker-data/ (NAS: 192.168.86.30:/volume1/Docker)
├── configs/          # Application configs
├── databases/        # Database data directories
├── media/            # Frigate recordings
├── traefik/          # Let's Encrypt certs
└── compose/          # Standalone compose files per node
    ├── hf-pdocker-01/{frigate,frigate-notify,clip-service,matrix,smarthome,guacamole,ai-agents,traefik,adguard}/
    ├── hf-pdocker-02/{unifi,gitea,semaphore,wikijs,vaultwarden,actual-budget,mcpo,trivy,watchtower}/
    └── bart/{media,openvas,watchtower}/
```

**NFS mount (all nodes):** `192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3,nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s 0 0`
**Note (hf-pdocker-01):** Both NFS mounts hardened with `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` on 2026-05-20 to prevent boot-time race (network not up when NFS tried to mount, cascading ~10 container failures).

**NFS + Databases:** Database containers on NFS require `user: "1024:100"` to avoid `chown` failures from NFS root_squash. UID 1024 is the NAS anonymous user.

### Repository File Structure

```
stacks/              # LEGACY Swarm stack definitions (Swarm dismantled 2026-04-01, kept for reference)

compose/             # Standalone Compose definitions (deployed per-node)
├── hf-pdocker-01/   # Core infrastructure + Frigate
├── hf-pdocker-02/   # Dev tools & utilities
│   ├── {service}/docker-compose.yml
│   └── {service}/.env
└── bart/            # Media stack & OpenVAS
```

### Deployment Workflow

**All services (compose/):**
1. Edit compose file in `compose/{node}/{service}/docker-compose.yml`
2. SCP to NAS path or node, then: `cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d`

### Service Architecture Patterns

**Network Patterns:**
- **Compose services**: Bridge networks for service isolation
- **Public services** (Traefik): Traefik labels + dynamic config files in `/mnt/docker-data/traefik/dynamic/`

**Database Patterns:**
- PostgreSQL: `postgres:15-alpine` or `postgres:16-alpine`
- MySQL: `mysql:8.0` (Guacamole, Semaphore)
- MongoDB: `mongo:7.0` (Unifi)
- **Always** add `com.centurylinklabs.watchtower.enable=false` label
- **Always** add `user: "1024:100"` for NFS-backed database volumes

**Volume Patterns:**
- Application data: `/mnt/docker-data/configs/{service-name}/`
- Database data: `/mnt/docker-data/databases/{service-name}-db/`
- Traefik certs: `/mnt/docker-data/traefik/letsencrypt/`

## Key Services & Access

### Public Services (via Traefik on hf-pdocker-01)

- Matrix: `mess.hobbs.farm`
- Guacamole: `solo.hobbs.farm`
- Home Assistant: `yoda.hobbs.farm`
- Vaultwarden: `vault.hobbs.farm`
- Gitea: `git.hobbs.farm`
- FRIDAY Voice: `coach.hobbs.farm` (replaced VoiceCoach 2026-07-05; runs on .150 :8320) — **DOWN (502) as of 2026-08-05, intentionally left down.** .150 Docker Desktop is offline so FRIDAY isn't running; a 2026-08-02 stopgap had repointed the Traefik route to a .192 voicecoach-web failover (`:8085`), which was removed in the 2026-08-05 VoiceCoach teardown → route now points at a dead backend. To restore: start Docker Desktop on .150, up friday-voice, repoint `friday-voice.yml` back to `.150:8320`. See memory `friday-voice`.
- MAS (Matrix auth): `auth.hobbs.farm` -- OAuth2/OIDC issuer. **If this is down, nobody can log into Matrix** (existing tokens keep working).
- Rongorongo: `rongorongo.hobbs.farm`
- LinkedStorm: `linkedstorm.hobbs.farm`

### Internal Services by Node

**hf-pdocker-01 (192.168.86.192):**
- Frigate: :5000
- Frigate Alerts (v2): :5199 (web UI + REST API)
- Home Assistant: :8123
- ESPHome: :6052
- Mosquitto: :1883
- Matrix/Synapse: :8008
- MAS: :8020 (web/oauth/compat), :8021 (health only -- /health is NOT on :8020)
- Well-known (Matrix federation): :8010
- Guacamole: :8199
- Traefik Dashboard: :8090 (HTTP: :80, HTTPS: :443)
- Grafana: :3001
- Prometheus: :9095
- Alertmanager: :9093
- InfluxDB: :8086
- Blackbox Exporter: :9115
- matrix-alertmanager-receiver: :9094
- AI Agents (margaret :18790, spirit :18795 [infra ops — monitors #notifications, auto-remediates alerts]) -- LAN-accessible (bind=lan, 0.0.0.0)
- Farm Manager: :8888 (dashboard + API)
- Farm Agent: :8889
- Rongorongo: :8181 (rongorongo-web nginx; reverse-proxies /api/* to rongorongo-api Flask backend on the internal `rongorongo` bridge net). State + images at /mnt/docker-data/configs/rongorongo/data/. Public URL gated by Traefik basicAuth (creds in Vaultwarden).
- LinkedStorm: :8300 (pipeline/VAD+STT+speaker-ID), :8310 (API/dashboard), cron (no port). Public: linkedstorm.hobbs.farm -> :8310. Source: /opt/linkedstorm (rsync from WSL). Data: /mnt/docker-data/configs/linkedstorm/. STT via Speaches GPU machine :8000, LLM via Ollama :11434 (qwen2.5:14b).
- ~~VoiceCoach~~ -- replaced by FRIDAY Voice on .150 (see GPU Machine section + memory `friday-voice`). **NOTE: the on-.192 stacks were only *stopped* in July, never torn down — the 2026-08-05 power-cycle reboot brought all 7 containers (~3GB RAM, incl. speechbrain 1.6GB) back. Actually `docker compose down`'d 2026-08-05** on both compose projects `voicecoach` + `voicecoach-v2` (dirs `/mnt/docker-data/compose/hf-pdocker-01/voicecoach{,-v2}` kept for reference). Containers + networks removed; won't restart on reboot now. Do NOT `docker compose up` those dirs.

**hf-pdocker-02 (192.168.86.100):**
- Farm Agent: :8889
- UniFi OS Server: :11443 (HTTPS, container 443)
- Gitea: :3003 (web), :2222 (SSH)
- Semaphore: :3004
- Wiki.js: :3002
- Vaultwarden: :26226
- MCPO: :8001
- Trivy: :4954
- CouchDB: :5984 -- Obsidian LiveSync backend. DB `secondbrain`; admin creds in Vaultwarden ("CouchDB - Obsidian LiveSync"). LAN-only (no Traefik route). Data on LOCAL disk `/var/lib/couchdb` (NFS breaks CouchDB file locking). **`local.ini` must be mounted READ-WRITE** -- with `:ro` CouchDB exits 1 and logs nothing. Auth required on everything except `/_up` (left open for the blackbox probe). Vault repo: git.hobbs.farm/jeremy/second-brain (private); vault lives at `C:\Users\Jeremy\second-brain` on the Windows box.

**bart (192.168.86.167):**
- Farm Agent: :8889
- Jellyfin: :8096
- Sonarr: :8989
- Radarr: :7878
- SABnzbd: :8082
- NZBHydra2: :5076
- Jellyseerr: :5055

**GPU Machine (192.168.86.150, RTX 3090):**
- **NOTE (2026-07-05):** .150 is actually the user's **Windows workstation** (runs the WSL/Claude Code session), not a headless Linux server. It runs **Docker Desktop** (WSL2 backend, RTX 3090 GPU passthrough working) alongside **UniFi OS Server** (Podman). Docker Desktop panics if UniFi/Podman claims `\\.\pipe\docker_engine` first — start Docker Desktop before UniFi OS Server. See memory `docker-desktop-unifi-podman-conflict`.
- **FRIDAY Voice replaced VoiceCoach (2026-07-05).** VoiceCoach fully decommissioned (v1+v2 removed from hf-pdocker-01; voice.hobbs.farm route removed). See memory `friday-voice`.
- [COMPOSE] **FRIDAY Voice stack (all on .150 Docker Desktop, via `docker.exe -c desktop-linux`):**
  - **voicecoach-speaches-gpu**: :8000 — GPU Whisper large-v3-turbo (STT) + Kokoro-82M af_nova (TTS), CUDA float16. Compose: `compose/gpu-machine/voicecoach-speaches/`.
  - **hermes**: `nousresearch/hermes-agent:latest` gateway — FRIDAY's reasoning agent. Settings transferred from WSL `~/.hermes` into named volume `hermes-data`. Compose: `compose/gpu-machine/hermes/`.
  - **friday-voice**: :8320 — push-to-talk console. Calls speaches (`host.docker.internal:8000`) + hermes (`docker exec hermes hermes -Q` via wrapper). Public: **coach.hobbs.farm** → :8320. Compose: `compose/gpu-machine/friday-voice/`.
- Ollama: :11434 (qwen2.5:14b, Hermes-4.3-36B-GGUF, others)

### Port Conflict Notes

Port conflicts already resolved:
- Port 8000: Freed (MCPO moved to 8001, Portainer moved elsewhere)
- Port 3000: freed 2026-07-13 (was AdGuard Home, decommissioned; Gitea remains on 3003)
- Port 3001: Grafana (Semaphore moved to 3004)
- Port 5000: Frigate uses host networking (registry container stopped)
- AI Agent ports re-spaced (2026-03-15): openclaw 2026.3.13 uses gateway+2 and gateway+3 for browser/chrome-relay. Agents spaced 5 apart: margaret=18790, spirit=18795

## Critical Configuration Preservation

### Encryption Keys & Passwords

- Semaphore: `SEMAPHORE_ACCESS_KEY_ENCRYPTION` encrypts sensitive data
- **Always** check for existing encryption keys before generating new ones

### Database Version Matching

Match the PostgreSQL/MySQL version to existing data directory format. Check logs for version mismatch errors.

## Post-Change Checklist (MANDATORY)

After ANY infrastructure change (deploy, remove, reconfigure, add service, change ports, modify Traefik routes), you MUST complete ALL of these before reporting done:

1. **CLAUDE.md** — Update service lists, ports, node assignments if changed
2. **MEMORY.md** — Add/update relevant memory entries
3. **Prometheus probes** — Add/remove blackbox targets, hot-reload Prometheus
4. **Wiki** — Launch documentation-keeper agent to update wiki pages (run in background)
5. **Farm Manager links page** — If a service was added/removed or its URL/port changed: edit the `serverLinks` array in `/opt/farm-manager/server/static/app.js` on hf-pdocker-01, then rebuild + redeploy: `docker build -f /opt/farm-manager/Dockerfile.server -t farm-manager:latest /opt/farm-manager && cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d` (links are baked into the image, not mounted)

Do NOT wait for the user to ask. Do NOT skip "because the change was small." Every change gets documented.

## Common Operations

### Deploy/Update a Compose Service

1. Edit `compose/{node}/{service}/docker-compose.yml`
2. SSH to the target node and run:
   ```bash
   cd /mnt/docker-data/compose/{node}/{service}
   docker compose up -d
   ```

### Troubleshoot

```bash
# On target node:
docker compose -f /mnt/docker-data/compose/{node}/{service}/docker-compose.yml logs --tail 100
docker ps --format "table {{.Names}}\t{{.Status}}"
```

### Check Cluster Health

```bash
# On each node:
docker ps                         # Local containers
```

## Data Management

### Storage

All persistent data on NAS at `/mnt/docker-data/`:
- `configs/` - Application configurations
- `databases/` - Database volumes
- `media/` - Frigate recordings
- `traefik/` - SSL certificates

### NAS Dependency

**Risk**: NAS failure = all services fail (single point of failure)
- NFS mount uses `soft,timeo=300` for graceful timeout
- Synology should have "Restart automatically after power failure" enabled
- Original data backups remain on local `/data/configs/` on hf-pdocker-01

## Specialized Agents

### vault-credential-manager
**Use for:** Storing/retrieving credentials from Vaultwarden

### documentation-keeper
**Use for:** Creating/updating documentation in Wiki.js

## Known Issues & Workarounds

1. **NFS root_squash**: Database containers need `user: "1024:100"` to avoid chown failures on NAS volumes
2. **Telegraf Docker Input**: Disabled due to Docker socket permission issues across nodes
3. **Watchtower API Version**: Requires `DOCKER_API_VERSION=1.44`
4. **InfluxDB Connection**: Telegraf may show timeout errors if InfluxDB is not running
5. **Frigate cameras**: Cameras on 192.168.44.x subnet may show RTSP timeouts (check camera network)
6. **Registry container**: Stopped on hf-pdocker-01 to free port 5000 for Frigate
7. **systemd-resolved DNSStubListener**: disabled on hf-pdocker-01 and hf-pdocker-02 via `/etc/systemd/resolved.conf.d/nostub.conf` (`DNSStubListener=no`) — originally for AdGuard Home's port 53. AdGuard was decommissioned 2026-07-13; the stub-listener change is harmless and left in place (re-enable only if something needs 127.0.0.53).
8. **NFS boot-mount race (hf-pdocker-01)**: NFS mounts failed at boot (network not ready in time). Fixed 2026-05-20 by adding `nofail,x-systemd.after=network-online.target,x-systemd.mount-timeout=60s` to both fstab entries. If mounts are missing after reboot: `sudo mount /mnt/docker-data && sudo mount /mnt/infrastructure`.
9. **Guacamole schema**: App image pinned to `guacamole/guacamole:1.6.0` + `guacd:1.6.0` with Watchtower disabled. DB schema was reset to fresh 1.6.0 on 2026-05-20 (old schema was non-standard). Recreate connections after any DB reset.
10. **Gitea image**: Pinned to `gitea/gitea:1.26.1` with Watchtower disabled (2026-05-20). Watchtower previously pulled broken `:latest` images causing missing git binary errors.
11. **Home Assistant reverse proxy**: Config at `/mnt/docker-data/configs/homeassistant/configuration.yaml` (no hyphen). Must have `http: use_x_forwarded_for: true` + `trusted_proxies: [172.16.0.0/12, 127.0.0.1]` or proxied requests return 400.
12. **hf-pdocker-01 memory pressure (2026-07-05, retrimmed 2026-07-11)**: Node runs hot (~15-18/23GB, chronic swap). Mitigations applied: `vm.swappiness` lowered 60→10 (persisted in `/etc/sysctl.d/99-swappiness.conf`), and VoiceCoach (v1+v2, ~10 containers) removed (replaced by FRIDAY Voice on .150) — **but see correction: the VoiceCoach compose stacks on .192 were only *stopped* in July, not `down`'d, and reappeared (~3GB) on the 2026-08-05 reboot; properly torn down 2026-08-05 (`docker compose down` on `voicecoach` + `voicecoach-v2`), node mem 9.0→6.7GB used.** Frigate is the main consumer. **2026-07-11:** Frigate's built-in AI features (`semantic_search: large`, `face_recognition`, `lpr`, `classification.bird`) were found *enabled* (contradicting the intended baseline) and re-disabled — dropped the embeddings CLIP model ~1.2GB→168MB and cut CPU inference; host mem 18→~9GB, swap 3.9→1.3GB right after. Config `/mnt/docker-data/configs/frigate/config/config.yml` (backup `.bak.20260711-aitrim`). Recognition is instead handled externally by frigate-notify v2 (clip-service was removed 2026-07-13 — frigate-notify's config never referenced it). **2026-07-12 — WHOLE-NODE CRASH:** .192 froze hard (~14:15, sshd/HA/DNS all dead, required hard reboot). Cause: Frigate `memswap_limit` had drifted to **16G** vs `mem_limit` 12G, letting it spill ~4GB into host swap; with the ffmpeg leak pinning it at 12G all day (cgroup OOM every ~45min), the host swap-thrashed to death. **Fix: node compose `memswap_limit: 12G` (= mem_limit) so Frigate can never use host swap** — a hit now trims its own ffmpeg (contained cgroup OOM), not the host. Repo already had 12G; node had drifted (never redeployed). NOTE: recording ffmpeg still leaks ~0.8GB/h atop ~6GB baseline (fills 12G in ~7.5h); twice-daily 4am+4pm restart-frigate cron mitigates but doesn't fully beat it — consider every-6h restart + fixing broken cameras (gate_camera, garage_2). Always keep `memswap_limit == mem_limit` on this node.
13. **Docker Desktop vs UniFi OS Server pipe conflict (Windows box .150, 2026-07-05)**: Both fight over `\\.\pipe\docker_engine`; Docker Desktop's backend nil-panics ("backend exited before becoming ready") if Podman wins. Fix: let Docker Desktop (auto-starts at login) claim the pipe first, launch UniFi OS Server manually after. Full detail in memory `docker-desktop-unifi-podman-conflict`.
14. **MySQL InnoDB corruption on NFS**: `semaphore-db` (mysql:8.0 on NFS) hit system-tablespace page corruption and crash-looped 3789× (2026-07-11), same class as the Guacamole-on-NFS issue (#9). Recovered in-place via `--innodb-force-recovery=1` → mysqldump → fresh volume → reimport (DB was empty, no data lost); admin recreated with `semaphore user add`. Any MySQL DB on NFS is at risk — prefer LOCAL storage like guacamole-db. See memory `semaphore-db-nfs-corruption`.
15. **litellm hangs after DB blip**: litellm does NOT self-recover from a transient `litellm-db` disconnect — it loops on "Prisma DB reconnect failed" forever; needs manual `docker restart litellm`. Its healthcheck was fixed from `curl` (absent from image, permanent false-unhealthy) to a python urllib check (2026-07-11).
16. **Stale NFS locks after unclean reboot/power loss (2026-07-12 power event)**: NFS file locks held by pre-reboot client identities survive on the Synology and block relocking — gitea crash-looped 7,972× on its LevelDB queue lock, semaphore-db 975× on `ibdata1`, and nzbhydra2 sat unhealthy on its H2 database lock. Fix applied 2026-07-13: lock-prone data moved to LOCAL disk (gitea queues → `/var/lib/gitea-queues`, semaphore-db → `/var/lib/semaphore-db` on .100; sonarr config → `/var/lib/sonarr`, nzbhydra config → `/var/lib/nzbhydra` on bart; jellyfin config → `/var/lib/jellyfin` on bart 2026-07-19 after 60-100s "database is locked" stalls on jellyfin.db caused playback lockups; **synapse data → `/var/lib/synapse` on .192 2026-08-20**, pre-emptively — its 279MB SQLite homeserver.db was on NFS). **Rule: anything holding long-lived file locks (MySQL, SQLite, LevelDB) goes on local disk, never NFS.**
17. **Synapse rate-limited Alertmanager notifications**: default `rc_message` (0.2 msg/s) made every matrix-alertmanager-receiver send fail with M_LIMIT_EXCEEDED 429 — **no alerts reached Matrix for weeks** while outages accumulated (Onyx down 5 weeks, AdGuard dead 2 months, unnoticed). Fixed 2026-07-13: `rc_message: {per_second: 5, burst_count: 50}` in homeserver.yaml (backup `.bak-20260713-ratelimit`).
18. **AdGuard Home decommissioned 2026-07-13**: its conf dir had been empty since 2026-05-17 (container ran the first-launch wizard, served no DNS, no backup existed) and the LAN had been using router DNS ever since. Container, data, compose dir, and Prometheus probe all removed; ports 53/3000 freed on .192. LAN DNS = the router.
19. **Duplicate failover stacks removed from .100 and bart (2026-07-13)**: .100 ran a second monitoring stack (prometheus/grafana/alertmanager/influxdb/blackbox), an idle traefik (shared acme.json with .192 — ACME conflict risk), and a stray guacamole; bart ran a failover guacamole. All removed (~2GB RAM reclaimed). Compose dirs kept for reference; data volumes untouched.
20. **UniFi OS Server probe port**: `unifi-os-server` on .100 serves HTTPS on host port **11443** (container 443), not 8443. Blackbox probe fixed 2026-07-13. solo.hobbs.farm returns 401 (Traefik basicAuth) when healthy — probed via `http_2xx_or_401` module.
21. **OpenVAS pg_upgrade loop saturated bart's HDD (2026-07-15)**: `immauss/openvas:latest` looped on the PG 13→15 auto-upgrade for ~21h (158 restarts) — each cycle re-copied the 21GB DB, pinning sda at ~90% util/40% iowait and making Jellyfin (same disk) hang. The upgrade itself succeeded every time but the final `mv /data/database /data/database_13_old` failed with "Device or resource busy" because the openvas-db volume is mounted **at** `/data/database` (can't rename a mountpoint). Fixed by wiping the DB (`docker volume rm openvas_openvas-db`) and letting it initdb fresh on PG15 — scan history lost, feeds re-synced. **On the next major PG bump this will recur** unless the upgrade is done out-of-band (content-level swap) or the volume is remounted at `/data`. General rule: a service "hanging" on bart usually means another container is eating the single HDD — check `iostat -x`/`pidstat -d` first. **OpenVAS fully decommissioned 2026-07-19** (moot unless redeployed).
22. **Jellyfin SQLite-on-NFS lockups (fixed 2026-07-19)**: jellyfin.db (WAL mode) on `/mnt/docker-data/configs/jellyfin` hit escalating `SQLite Error 5: 'database is locked'` (22→33→125/day), DB commands blocking 60-102s → healthcheck flaps + playback stalls ("Unexpected end of request content" on /Sessions/Playing/Progress). Config moved to LOCAL `/var/lib/jellyfin` (old copy left on NFS as backup). Same class as Known Issue #16.


23. **Matrix registration was wide open to the internet (fixed 2026-08-20)**: `mess.hobbs.farm` is publicly reachable (Traefik, port 443, federation + client API on :8008) and `homeserver.yaml` had `enable_registration: true` + `enable_registration_without_verification: true` — the registration flow was a bare `m.login.dummy`, i.e. **anyone on the internet could create an account on hobbs.farm with no email, captcha, or token**. No abuse had occurred (all 18 accounts are known/expected; newest legit account was @fry 2026-07-05). Fixed by setting both to `false` and `restrict_public_rooms_to_local_users: true` (was `false`, which published the public room directory to any remote server). Backup: `/var/lib/synapse/homeserver.yaml.bak-20260820-registration`. New accounts are still creatable on the box via `docker exec synapse register_new_matrix_user -k <registration_shared_secret>`.
24. **Synapse database history (SQLite -> local disk -> PostgreSQL, all 2026-08-20)**: Synapse ran on **SQLite on NFS** while CLAUDE.md claimed it used a `synapse-db` postgres that in fact had **0 tables**. First the data was moved to LOCAL `/var/lib/synapse` and the empty postgres deleted; then, because `syn2mas` **only reads PostgreSQL** ("The migration tool only supports reading from PostgreSQL for the Synapse database as well"), a real `synapse-db` was created and the DB ported with `synapse_port_db` (20 users / 93,888 events / 65 devices, verified identical; 2m35s downtime). The PG database MUST have `LC_COLLATE=C` -- verify with `select datcollate from pg_database where datname='synapse'` (PG16 removed the `SHOW lc_collate` GUC). The old SQLite file is still at `/var/lib/synapse/homeserver.db`, now inert; NFS copy at `/mnt/docker-data/configs/matrix/data` is an older backup -- **never remount either**.
25. **Synapse upgraded 1.148.0 -> 1.159.0 and moved to the element-hq registry (2026-08-20)**: it had been on `matrixdotorg/synapse:latest`, an **abandoned Docker Hub repo** whose `:latest` froze at 1.148.0 (image built 2026-02-24) — a moving tag that silently stopped moving on a publicly-exposed homeserver. Now pinned to `ghcr.io/element-hq/synapse:v1.159.0` (note the **`v` prefix** — `:1.159.0` without it 404s). Upgrade was clean: schema 93 -> 94, all background updates finished in ~5s, 0 errors, 19 users / 93,884 events intact, memory 112MB/1G. None of the v1.149-v1.159 breaking changes applied to us (MSC3861 removal — unused; `quarantined_media_changes` stream writer — workers only; the rest are Debian/Ubuntu packaging). Pre-upgrade backup: `/var/lib/synapse.bak-20260820-pre1159` (646MB) — delete once you're satisfied. **Pinned on purpose: bump it manually and deliberately, since the last moving tag rotted for 6 months unnoticed.**
26. **`SYNAPSE_SERVER_NAME` in the matrix compose disagrees with homeserver.yaml**: compose passes `SYNAPSE_SERVER_NAME=mess.hobbs.farm` but the real `server_name` is **`hobbs.farm`** (user IDs are `@user:hobbs.farm`). The env var is only read when the image *generates* a config, so it's inert today — but if `homeserver.yaml` ever went missing, the container would generate a fresh config with the wrong `server_name` AND a new signing key, which would break federation and every existing user ID. Left as-is (changing it has no effect while the config exists); just never let the container auto-generate..

27. **Matrix auth is delegated to MAS -- Synapse does NOT serve /login any more (2026-08-20)**: with `matrix_authentication_service.enabled: true`, Synapse answers `M_UNRECOGNIZED` on `/_matrix/client/*/login`. The **reverse proxy must route the three compat paths to MAS or nobody can sign in**: `login`, `logout`, `refresh` (verified: MAS returns 200/405 on those and 404 on `register`/`whoami`, which stay on Synapse). Implemented as the `mess-auth-compat` router in `/mnt/docker-data/traefik/dynamic/temp-routes.yml` with explicit `priority: 1000` so it beats the catch-all `mess` router. **Anything that talks to `192.168.86.192:8008` directly bypasses Traefik and therefore cannot log in with a password** -- token-based bots (spirit, frigate, alertbot) are unaffected because they never call /login, but `@clippyauto` (n8n) and `@pluto` historically re-login with a password (17 and 15 accumulated devices). They currently run on migrated tokens; if either ever needs a fresh password login, point it at `https://mess.hobbs.farm` instead of `:8008`.
28. **MAS image tags are NOT `v`-prefixed** -- `ghcr.io/element-hq/matrix-authentication-service:1.23.0` is correct, `:v1.23.0` does not exist. This is the **opposite** of Synapse (`ghcr.io/element-hq/synapse:v1.159.0`, where `:1.159.0` 404s). The MAS image is also **distroless** -- no shell, so a container healthcheck is impossible; liveness comes from the blackbox probe of `http://192.168.86.192:8021/health`. Note `/health` lives on the internal listener only: `https://auth.hobbs.farm/health` returns 404 by design, so the public probe targets `/.well-known/openid-configuration`.
29. **MAS config holds unrecoverable signing keys**: `/mnt/docker-data/configs/mas/config.yaml` (mode 640) contains the RSA/EC signing keys and the `matrix.secret` shared with Synapse. Regenerating them invalidates every session. It is **not yet backed up to Vaultwarden** because the vault master password is currently unknown (see memory `vault-master-password-stale`) -- resolve that and store it. Also note `http.issuer`/`public_base` (`https://auth.hobbs.farm/`) are baked into session state; changing the hostname later invalidates sessions.