Commit Graph

11 Commits

Author SHA1 Message Date
jeremy 45573cf9fc fix: nzbhydra2 stale NFS H2 lock -> local /var/lib/nzbhydra; add Farm Manager links page to post-change checklist
nzbhydra2 was the 4th victim of the 2026-07-12 power-event stale NFS locks
(H2 'file is locked' retry loop; the earlier mem bump was a red herring).
Farm Manager links updated (AdGuard/Actual Budget/Portainer removed, UniFi
-> :11443) and image rebuilt; links maintenance is now checklist item 5.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 20:39:14 -05:00
jeremy 5d7c8ea030 fix: recover from 2026-07-12 power-event fallout + efficiency cleanup
- gitea: LevelDB queues -> local /var/lib/gitea-queues (stale NFS lock, 7,972-restart loop)
- semaphore-db: data -> local /var/lib/semaphore-db (stale ibdata1 NFS lock, 975 restarts)
- sonarr: config -> local /var/lib/sonarr (SQLite-on-NFS migration crash)
- nzbhydra2: mem_limit 768M -> 1G (was pinned at limit, unhealthy)
- jellyfin: deploy pending VAAPI device passthrough (i965/renderD128)
- openvas: disable Watchtower (latest major pull forced pg_upgrade crash-loop)
- onyx: fully decommissioned (down since ~June 8 unnoticed)
- remove duplicate monitoring/traefik/guacamole stacks from .100, failover guacamole from bart
- synapse: raise rc_message (alertmanager receiver was 429-limited; alerts silently dropped for weeks)
- prometheus: UniFi probe 8443 -> 11443, solo.hobbs.farm via 401-tolerant module, drop onyx target
- frigate restart cron: unwritable /var/log path meant it never ran; now 4x daily to ~glyptic
- CLAUDE.md: document all of the above + AdGuard broken/unconfigured since 2026-05-17

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 19:20:10 -05:00
jeremy 27f6d95bf6 chore: sync accumulated infra changes and docs
Removals: openwebui and netclaw compose + Traefik routes (both decommissioned).
Updates: litellm, n8n, vaultwarden, ai-agents, media, traefik routes, CLAUDE.md
(nodes/ports/known-issues), agent definitions.
Additions: Rongorongo tree, LinkedStorm plan, rename-bart spec, vllm brief,
tls-options and security-headers Traefik config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 11:19:21 -05:00
jeremy 971fe8dda8 feat: standalone node-exporter + telegraf compose per node
Post-Swarm: node-exporter must run standalone with pid:host, since Swarm mode
blocked it and cross-node /proc namespaces contaminated swap metrics. Marks the
stacks/ definition superseded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 11:19:04 -05:00
jeremy 904d1ef397 fix: onyx nginx - use rewrite to strip /api/ prefix with variable proxy_pass
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 13:58:30 -05:00
jeremy a7dfbdeb79 fix: onyx nginx - use Docker DNS resolver to prevent stale upstream IPs
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 13:30:42 -05:00
jeremy 6233cf9e1d fix: disable Onyx telemetry (stops data posting to telemetry.onyx.app) 2026-05-22 09:00:46 -05:00
jeremy 0d5b781aef fix: onyx nginx - strip /api prefix when proxying to api_server
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-22 08:30:37 -05:00
jeremy 752b8230ed infra: clean redeploy of Onyx on bart with latest images
Full wipe and fresh start: removed gitea_patch/unstructured_patch
hacks from startup commands, upgraded all onyxdotapp images from
nightly-latest-20260331 pin to :latest, removed UNSTRUCTURED_SERVER_URL
(service decommissioned 2026-04-14), added Watchtower disable labels
to ALL services (was missing on web_server, model_servers, db, cache,
opensearch, vespa, nginx), added nginx.conf for reverse proxy.

Alembic ran 349 migrations cleanly on fresh DB. All 10 containers up,
0 restarts, HTTP 200 at :3005.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-20 07:12:03 -05:00
jeremy 04a18efc6a infra: pin onyx-backend to nightly-latest-20260331, disable Watchtower
DB alembic HEAD is 1d78c0ca7853 (remove_voice_provider_deleted_column).
20260331 is the exact build where that revision is HEAD — alembic runs
clean with no migrations applied. Later builds (20260412/latest) introduce
migrations d8cdfee5df80–503883791c39 that break api_server startup.

Watchtower disabled on api_server and background to prevent future
auto-upgrades from breaking alembic compatibility.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-12 15:00:42 -05:00
jeremy de1d9e1d93 infra: full IaC state after hf-pdocker-01 rebuild (2026-03-08)
Replaces initial Nomad/Consul/Terraform skeleton with current Docker-based
infrastructure across all three nodes (hf-pdocker-01, hf-pdocker-02, bart).

Key changes:
- CLAUDE.md: updated service distribution reflecting hf-pdocker-01 restored
- docs/plans/2026-03-08-hf-pdocker-01-rebuild.md: new rebuild plan doc
- compose/bart/: ai-agents, guacamole, matrix, media, openvas, smarthome, watchtower
- compose/hf-pdocker-02/: traefik, monitoring, adguard, gitea, semaphore, wikijs,
  unifi, vaultwarden, actual-budget, mcpo, netclaw, trivy, watchtower, n8n
- compose/hf-pdocker-01/: frigate, frigate-notify, farm-agent
- stacks/: all Swarm stack definitions (monitoring, traefik, smarthome, matrix,
  guacamole, media, ai-agents, frigate, gitea, semaphore, wikijs, unifi, etc.)
- .gitignore: exclude .env credential files and acme.json certs
- .env.example: placeholder credential files for all services
- .claude/agents/: gitea-backup and other infrastructure agent configs
- scripts/: frigate media cleanup and NAS sync scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-08 18:51:25 -05:00