Files
hobbs-iac/docs/plans/2026-03-08-hf-pdocker-01-phase2-migration.md
jeremy cd37bb9858 infra: hf-pdocker-01 Phase 2 migration — all services returned from failover
Node hf-pdocker-01 hardware fully repaired and rebuilt (2026-03-08). This
commit captures the completed Phase 2 migration where all failover services
are returned from hf-pdocker-02/bart back to hf-pdocker-01 as their permanent
home.

New compose stacks for hf-pdocker-01:
- adguard: AdGuard Home DNS/ad-blocking (:3000, :53)
- traefik: Reverse proxy with TLS (:80/:443/:8090), .env.example included
- monitoring: Grafana, Prometheus, Alertmanager, InfluxDB, Blackbox, matrix-receiver
- smarthome: Home Assistant (:8123), ESPHome (:6052), Mosquitto (:1883/:8883)
- matrix: Synapse homeserver (:8008) + synapse-db + well-known (:8010)
- guacamole: Guacamole (:8199) + guacd + guacamole-db (local MySQL)
- farm-manager: Farm Manager API (:8888)

Updated compose stacks for hf-pdocker-01:
- frigate-notify: MQTT/Matrix hosts reverted from bart (.167) to hf-pdocker-01 (.192)

Documentation:
- docs/plans/2026-03-08-hf-pdocker-01-phase2-migration.md (new)
- CLAUDE.md: Full service distribution update — failover labels removed,
  hf-pdocker-02/bart cleaned up to reflect only permanent services,
  NetClaw/Aaron added to hf-pdocker-02, DNSStubListener fix noted

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-08 19:58:03 -05:00

4.0 KiB

hf-pdocker-01 Phase 2 — Service Migration Back

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Migrate all failover services back to hf-pdocker-01 from hf-pdocker-02 and bart.

Architecture: Stop services on failover nodes, start on hf-pdocker-01, update IP references and routing. Order matters due to dependencies.

Dependency Chain:

  • Mosquitto must be up before Home Assistant and Frigate MQTT config update
  • Matrix/Synapse must be up before AI Agents
  • Traefik must be up before external routing works
  • AdGuard is DNS — move carefully to minimize resolution downtime

Task 1: Migrate Mosquitto (bart → hf-pdocker-01)

MQTT broker — Home Assistant and Frigate depend on it.

  1. Deploy Mosquitto on hf-pdocker-01 using compose from /mnt/docker-data/compose/bart/smarthome/ (extract mosquitto service) or create standalone compose
  2. Verify MQTT is listening on hf-pdocker-01:1883
  3. Update Frigate config MQTT host back to 192.168.86.192 (localhost)
  4. Restart Frigate to pick up new MQTT
  5. Stop Mosquitto on bart

Task 2: Migrate Home Assistant + ESPHome (bart → hf-pdocker-01)

Smart home services — depend on Mosquitto.

  1. Deploy HA + ESPHome on hf-pdocker-01
  2. Update HA config: MQTT broker IP back to 192.168.86.192
  3. Verify HA web UI on :8123, ESPHome on :6052
  4. Stop HA + ESPHome on bart

Task 3: Migrate Matrix/Synapse (bart → hf-pdocker-01)

Matrix homeserver — AI agents and alertbot depend on it.

  1. Deploy Synapse + synapse-db + wellknown on hf-pdocker-01
  2. Verify Matrix federation and client API on :8008
  3. Update matrix-alertmanager-receiver config to point to new Synapse
  4. Stop Synapse stack on bart

Task 4: Migrate Guacamole (bart → hf-pdocker-01)

Remote desktop gateway — has local MySQL DB on bart (NFS lock workaround).

  1. Copy guacamole-db data from bart:/var/lib/guacamole-db to hf-pdocker-01 local storage
  2. Deploy Guacamole + guacd + guacamole-db on hf-pdocker-01
  3. Verify web UI on :8199
  4. Stop Guacamole stack on bart

Task 5: Migrate AI Agents (bart → hf-pdocker-01)

margaret, spirit, bob, clippy — depend on Matrix/Synapse.

  1. Build margaret-openclaw:latest image on hf-pdocker-01
  2. Update agent configs: Matrix homeserver URL back to 192.168.86.192:8008
  3. Rename Matrix account state dirs
  4. Deploy all 4 agents on hf-pdocker-01
  5. Verify Matrix connectivity
  6. Stop agents on bart

Task 6: Migrate AdGuard Home (hf-pdocker-02 → hf-pdocker-01)

DNS server — brief downtime expected during switchover.

  1. Deploy AdGuard on hf-pdocker-01 using NAS config
  2. Verify DNS resolution on :53 and web UI on :3000
  3. Stop AdGuard on hf-pdocker-02

Task 7: Migrate Traefik (hf-pdocker-02 → hf-pdocker-01)

Reverse proxy — handles all external HTTPS routing.

  1. Deploy Traefik on hf-pdocker-01 with NAS certs and dynamic configs
  2. Update dynamic configs for new service IPs (services now on .192 instead of .167)
  3. Verify HTTPS routing for mess/solo/yoda/vault/git.hobbs.farm
  4. Stop Traefik on hf-pdocker-02

Task 8: Migrate Monitoring Stack (hf-pdocker-02 → hf-pdocker-01)

Grafana, Prometheus, Alertmanager, InfluxDB, Blackbox, matrix-receiver.

  1. Deploy full monitoring stack on hf-pdocker-01
  2. Update Prometheus config with correct service IPs (most services now back on .192)
  3. Update Telegraf configs on all nodes to point InfluxDB to .192
  4. Verify Grafana on :3001, Prometheus on :9095
  5. Stop monitoring stack on hf-pdocker-02

Task 9: Migrate Farm Manager API (new deploy on hf-pdocker-01)

Was never failed over — needs fresh deploy.

  1. Clone farm-manager from Gitea, build server image
  2. Deploy Farm Manager API on :8888
  3. Update config.json with correct node IPs
  4. Verify dashboard and API

Task 10: Cleanup + Final Verification

  1. Remove stopped failover containers on bart and hf-pdocker-02
  2. Update all Prometheus blackbox targets for final service locations
  3. Hot-reload Prometheus
  4. Update Traefik dynamic configs
  5. Verify all services healthy
  6. Update CLAUDE.md, MEMORY.md, wiki