Commit Graph

21 Commits

Author SHA1 Message Date
jeremy beb727528a docs: UniFi controller consolidation, STP fix, NFS recovery
- Remove duplicate unifi-os-server container from .100; sole UniFi
  controller is now the .150 Podman instance (UI :11443, inform :8080).
  Data preserved; compose dir kept.
- Document STP root-cause/fix for wifi client drops: version mismatch
  (stp vs rstp) + default priorities -> root flapping; fixed by RSTP
  everywhere + .159 pinned root @4096.
- Known Issue #20: correct the stale .100:11443 blackbox probe note -
  the probe DID exist in the live config and was removed 2026-10-06
  (backup saved) + Prometheus reloaded; UnifiDeviceDown rule is inert.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-10-06 19:06:05 -05:00
jeremy 9a81138cc9 docs: QR sign-in confirmed working end-to-end
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 16:36:27 -05:00
jeremy 84021278f9 docs: MAS deployment - CLAUDE.md service lists and known issues 24,27,28,29
- Synapse DB history rewritten (SQLite -> local -> PostgreSQL, and why)
- #27: Synapse no longer serves /login; Traefik compat routing is load-bearing
- #28: MAS tags are not v-prefixed; distroless; /health only on :8021
- #29: MAS signing keys need a vault home (vault currently locked out)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-20 15:21:21 -05:00
jeremy bfb9b03a26 fix: move jellyfin config to local /var/lib/jellyfin (NFS SQLite lock stalls); decommission OpenVAS
Jellyfin lockups root-caused to jellyfin.db (WAL mode) on NFS: SQLite
"database is locked" errors escalated 22->33->125/day, with DB commands
blocking 60-102s, causing healthcheck flaps and playback stalls. Config
rsynced from /mnt/docker-data/configs/jellyfin to local /var/lib/jellyfin
on bart; healthy after redeploy. jellyfin:2026060106 v12 image pin
unchanged. Old NFS copy left in place as backup.

OpenVAS fully removed from bart per user (~39GB reclaimed: containers,
volumes incl. PG DB, image). Prometheus probe and Farm Manager link
removed. Compose dir kept for reference; Known Issue #21 marked moot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 18:21:20 -05:00
jeremy 8a051939ed docs: OpenVAS pg_upgrade mountpoint loop (Known Issue #21) — DB wiped fresh 2026-07-15
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-14 20:03:31 -05:00
jeremy 66c65d1b4e fix: repin jellyfin to v12 tag 2026060106 (DB is v12 schema)
The 2026-07-13 media redeploy overwrote a node-side pin with the repo's
:latest (10.11.x), downgrading jellyfin under its v12-migrated DB and
breaking Resume/home queries ('no such column: b.ExtraIds'). Repinned;
do not unpin until official stable >= 12.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 21:24:21 -05:00
jeremy 45573cf9fc fix: nzbhydra2 stale NFS H2 lock -> local /var/lib/nzbhydra; add Farm Manager links page to post-change checklist
nzbhydra2 was the 4th victim of the 2026-07-12 power-event stale NFS locks
(H2 'file is locked' retry loop; the earlier mem bump was a red herring).
Farm Manager links updated (AdGuard/Actual Budget/Portainer removed, UniFi
-> :11443) and image rebuilt; links maintenance is now checklist item 5.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 20:39:14 -05:00
jeremy d35af1ddad chore: decommission clip-service (.192) and Actual Budget (.100)
clip-service: container+volumes+compose removed, ~2GB RAM freed; frigate-notify
config never referenced it. Actual Budget: container+compose removed, data kept
at /mnt/docker-data/configs/actual-budget. Prometheus probes for both removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 20:24:29 -05:00
jeremy ed5f330d2c chore: decommission AdGuard Home (dead since 2026-05-17, LAN uses router DNS)
Container, data, compose dir, and Prometheus probe removed from .192.
Ports 53 and 3000 freed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 20:20:37 -05:00
jeremy 5d7c8ea030 fix: recover from 2026-07-12 power-event fallout + efficiency cleanup
- gitea: LevelDB queues -> local /var/lib/gitea-queues (stale NFS lock, 7,972-restart loop)
- semaphore-db: data -> local /var/lib/semaphore-db (stale ibdata1 NFS lock, 975 restarts)
- sonarr: config -> local /var/lib/sonarr (SQLite-on-NFS migration crash)
- nzbhydra2: mem_limit 768M -> 1G (was pinned at limit, unhealthy)
- jellyfin: deploy pending VAAPI device passthrough (i965/renderD128)
- openvas: disable Watchtower (latest major pull forced pg_upgrade crash-loop)
- onyx: fully decommissioned (down since ~June 8 unnoticed)
- remove duplicate monitoring/traefik/guacamole stacks from .100, failover guacamole from bart
- synapse: raise rc_message (alertmanager receiver was 429-limited; alerts silently dropped for weeks)
- prometheus: UniFi probe 8443 -> 11443, solo.hobbs.farm via 401-tolerant module, drop onyx target
- frigate restart cron: unwritable /var/log path meant it never ran; now 4x daily to ~glyptic
- CLAUDE.md: document all of the above + AdGuard broken/unconfigured since 2026-05-17

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 19:20:10 -05:00
jeremy 81a7464dd8 docs: record CouchDB/Obsidian LiveSync in CLAUDE.md
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 12:04:02 -05:00
jeremy f0e2918bd1 revert: remove LifeOS deployment
Evaluated and torn down at the user's request. Container, image, source
(/opt/lifeos), SQLite data (/var/lib/lifeos) and compose dir removed from
hf-pdocker-02; Prometheus probe removed and both instances reloaded; port
8095 released.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 11:43:14 -05:00
jeremy 27f6d95bf6 chore: sync accumulated infra changes and docs
Removals: openwebui and netclaw compose + Traefik routes (both decommissioned).
Updates: litellm, n8n, vaultwarden, ai-agents, media, traefik routes, CLAUDE.md
(nodes/ports/known-issues), agent definitions.
Additions: Rongorongo tree, LinkedStorm plan, rename-bart spec, vllm brief,
tls-options and security-headers Traefik config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 11:19:21 -05:00
jeremy 6f485ff6e5 docs: update CLAUDE.md — Coral USB TPU, Onyx redeploy, infra fixes 2026-05-20
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-20 08:06:05 -05:00
jeremy a908973264 infra: rongorongo state-sync backend + Traefik basicAuth
Add a Flask backend service for cross-device state persistence, storing
JSON state on an NFS volume at /mnt/docker-data/configs/rongorongo/data.
Frontend Dockerfile/nginx.conf reorganised into frontend/ subdirectory;
frontend API calls now route to /api/* handled by the backend container
over a dedicated bridge network. Traefik basicAuth middleware added to
the public URL with the bcrypt hash deliberately kept out of git
(REPLACE_AT_DEPLOY placeholder in rongorongo.yml). Prometheus probe moved
from https-external to internal http://192.168.86.192:8181/health to
bypass the new auth wall.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-07 18:51:47 -05:00
jeremy 795908e1ef infra: deploy Rongorongo decipherment engine to hf-pdocker-01
Standalone Compose service running a locally-built nginx image that
serves a static Anthropic-API frontend. Traefik dynamic route added
for public HTTPS access. Blackbox probe added to prometheus.yml for
uptime monitoring. CLAUDE.md updated with service entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-06 13:10:47 -05:00
jeremy 9943fd4410 infra: repurpose Spirit as autonomous infrastructure ops agent
Spirit now monitors #notifications Matrix room for Prometheus alerts,
auto-investigates issues, takes corrective action (container restarts,
disk cleanup, etc.), and documents incidents in Wiki.js.

Changes:
- Custom spirit-ops Docker image (Docker CLI + SSH + diagnostic tools)
- Docker socket mount for local container management
- SSH keys for remote access to hf-pdocker-02 and bart
- Wiki.js API integration for incident documentation
- Prometheus probe for Spirit gateway health
- Safety guardrails: protected services, rate limiting, escalation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 10:36:20 -05:00
jeremy 59f529e3a7 infra: deploy Open WebUI on hf-pdocker-01 with Ollama + Claude backends
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-11 14:13:16 -05:00
jeremy cd37bb9858 infra: hf-pdocker-01 Phase 2 migration — all services returned from failover
Node hf-pdocker-01 hardware fully repaired and rebuilt (2026-03-08). This
commit captures the completed Phase 2 migration where all failover services
are returned from hf-pdocker-02/bart back to hf-pdocker-01 as their permanent
home.

New compose stacks for hf-pdocker-01:
- adguard: AdGuard Home DNS/ad-blocking (:3000, :53)
- traefik: Reverse proxy with TLS (:80/:443/:8090), .env.example included
- monitoring: Grafana, Prometheus, Alertmanager, InfluxDB, Blackbox, matrix-receiver
- smarthome: Home Assistant (:8123), ESPHome (:6052), Mosquitto (:1883/:8883)
- matrix: Synapse homeserver (:8008) + synapse-db + well-known (:8010)
- guacamole: Guacamole (:8199) + guacd + guacamole-db (local MySQL)
- farm-manager: Farm Manager API (:8888)

Updated compose stacks for hf-pdocker-01:
- frigate-notify: MQTT/Matrix hosts reverted from bart (.167) to hf-pdocker-01 (.192)

Documentation:
- docs/plans/2026-03-08-hf-pdocker-01-phase2-migration.md (new)
- CLAUDE.md: Full service distribution update — failover labels removed,
  hf-pdocker-02/bart cleaned up to reflect only permanent services,
  NetClaw/Aaron added to hf-pdocker-02, DNSStubListener fix noted

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-08 19:58:03 -05:00
jeremy 26d559feb5 Phase 2 Step 2: Migrate Home Assistant + ESPHome back to hf-pdocker-01
- HA and ESPHome now running on hf-pdocker-01 via smarthome compose stack
- Updated core.config_entries: MQTT broker, go2rtc, Frigate URLs all point to 192.168.86.192
- Removed stale .ha_run.lock before startup
- Stopped and removed HA + ESPHome containers from bart
- All smarthome services (mosquitto, homeassistant, esphome) now consolidated on hf-pdocker-01
2026-03-08 19:03:19 -05:00
jeremy de1d9e1d93 infra: full IaC state after hf-pdocker-01 rebuild (2026-03-08)
Replaces initial Nomad/Consul/Terraform skeleton with current Docker-based
infrastructure across all three nodes (hf-pdocker-01, hf-pdocker-02, bart).

Key changes:
- CLAUDE.md: updated service distribution reflecting hf-pdocker-01 restored
- docs/plans/2026-03-08-hf-pdocker-01-rebuild.md: new rebuild plan doc
- compose/bart/: ai-agents, guacamole, matrix, media, openvas, smarthome, watchtower
- compose/hf-pdocker-02/: traefik, monitoring, adguard, gitea, semaphore, wikijs,
  unifi, vaultwarden, actual-budget, mcpo, netclaw, trivy, watchtower, n8n
- compose/hf-pdocker-01/: frigate, frigate-notify, farm-agent
- stacks/: all Swarm stack definitions (monitoring, traefik, smarthome, matrix,
  guacamole, media, ai-agents, frigate, gitea, semaphore, wikijs, unifi, etc.)
- .gitignore: exclude .env credential files and acme.json certs
- .env.example: placeholder credential files for all services
- .claude/agents/: gitea-backup and other infrastructure agent configs
- scripts/: frigate media cleanup and NAS sync scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-08 18:51:25 -05:00