hobbs-iac/docs/plans/2026-03-08-hf-pdocker-01-rebuild.md

hf-pdocker-01 Rebuild — Phase 1 Implementation Plan

For Claude: REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

Goal: Rebuild hf-pdocker-01 from fresh Ubuntu 24.04 to a fully operational Swarm manager with Frigate NVR, Farm Agent, and Frigate-Notify v2.

Architecture: Fresh Ubuntu 24.04 on NVMe, Docker CE, NFS-backed storage from Synology NAS, 5TB USB disk for Frigate recordings. Docker Swarm re-initialized as single manager with hf-pdocker-02 and bart rejoined as workers. Services deployed via Compose files already on NAS.

Tech Stack: Ubuntu 24.04, Docker CE, Docker Swarm, NFS v3, Intel VA-API/OpenVINO, Python 3.12 (containers)

Access: SSH to hf-pdocker-01 (192.168.86.192), user glyptic, password &Rip8a115

Important: All tasks execute via SSH from the WSL environment. Use the docker-swarm-ops agent for all remote operations.

---

Task 1: Install System Prerequisites

Target: hf-pdocker-01 (192.168.86.192)

Step 1: Install base packages

sudo apt-get update
sudo apt-get install -y nfs-common curl jq git rsync

Step 2: Install Docker CE

# Add Docker GPG key and repo
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Step 3: Add glyptic to docker group

sudo usermod -aG docker glyptic

Step 4: Install Intel VA-API drivers for Frigate OpenVINO

sudo apt-get install -y intel-media-va-driver-non-free vainfo

Step 5: Verify

docker --version         # Expect: Docker version 28.x
docker compose version   # Expect: Docker Compose version v2.x
vainfo                   # Expect: VA-API device listed (iHD driver, HD 630)
ls /dev/dri/             # Expect: card0, renderD128

---

Task 2: Configure System Settings

Target: hf-pdocker-01 (192.168.86.192)

Step 1: Set vm.swappiness to 10 (cluster standard)

echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.conf
sudo sysctl vm.swappiness=10

Step 2: Verify

sysctl vm.swappiness  # Expect: vm.swappiness = 10

---

Task 3: Configure NFS Mounts

Target: hf-pdocker-01 (192.168.86.192)

Step 1: Create mount points

sudo mkdir -p /mnt/docker-data /mnt/infrastructure

Step 2: Add NFS entries to fstab

echo '192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3 0 0' | sudo tee -a /etc/fstab
echo '192.168.86.30:/volume1/Infrastructure /mnt/infrastructure nfs rw,hard,intr,_netdev 0 0' | sudo tee -a /etc/fstab

Step 3: Mount and verify

sudo mount -a
ls /mnt/docker-data/configs/    # Expect: frigate, frigate-notify, etc.
ls /mnt/infrastructure/margaret/  # Expect: openclaw.json, credentials/, etc.

---

Task 4: Mount 5TB Frigate Recordings Disk

Target: hf-pdocker-01 (192.168.86.192)

Step 1: Identify the disk and verify existing data

sudo blkid /dev/sda1              # Check filesystem type (expect ext4)
sudo mkdir -p /mnt/frigate-recordings
sudo mount /dev/sda1 /mnt/frigate-recordings
ls /mnt/frigate-recordings/       # Expect: frigate recording directories

Step 2: Add to fstab using UUID for reliability

# Get UUID
UUID=$(sudo blkid -s UUID -o value /dev/sda1)
echo "UUID=$UUID /mnt/frigate-recordings ext4 defaults,_netdev 0 2" | sudo tee -a /etc/fstab

Step 3: Verify

df -h /mnt/frigate-recordings  # Expect: ~4.5T disk

---

Task 5: Re-initialize Docker Swarm

Target: hf-pdocker-01 (192.168.86.192), then hf-pdocker-02 and bart

Step 1: Initialize Swarm on hf-pdocker-01

# On hf-pdocker-01:
docker swarm init --advertise-addr 192.168.86.192

Save the worker join token from the output.

Step 2: Clean up stale Swarm state on workers

# On hf-pdocker-02 (192.168.86.100):
docker swarm leave --force

# On bart (192.168.86.167):
docker swarm leave --force

Step 3: Rejoin workers

# On hf-pdocker-02:
docker swarm join --token <WORKER_TOKEN> 192.168.86.192:2377

# On bart:
docker swarm join --token <WORKER_TOKEN> 192.168.86.192:2377

Step 4: Verify cluster

# On hf-pdocker-01:
docker node ls   # Expect: 3 nodes, hf-pdocker-01 as Leader/Manager, others as Workers

Step 5: Recreate overlay networks

# On hf-pdocker-01:
docker network create --driver overlay --attachable traefik-public

---

Task 6: Deploy Frigate NVR

Target: hf-pdocker-01 (192.168.86.192)

Step 1: Update Frigate config MQTT address

The Frigate config at /mnt/docker-data/configs/frigate/config/config.yml has MQTT pointed at 192.168.86.192:1883 (the old hf-pdocker-01 address). Mosquitto is currently running on bart (192.168.86.167) as a failover service.

# On hf-pdocker-01, edit the config to point MQTT to bart:
sed -i 's/host: 192.168.86.192/host: 192.168.86.167/' /mnt/docker-data/configs/frigate/config/config.yml

Verify the change:

grep -A2 "^mqtt:" /mnt/docker-data/configs/frigate/config/config.yml
# Expect: host: 192.168.86.167

Step 2: Deploy Frigate

cd /mnt/docker-data/compose/hf-pdocker-01/frigate
docker compose up -d

Step 3: Verify

docker ps --filter name=frigate --format "table {{.Names}}\t{{.Status}}"  # Expect: Up
curl -s http://localhost:5000/api/version  # Expect: Frigate version response
curl -s http://localhost:5000/api/stats | jq '.detectors'  # Expect: openvino detector ~10ms inference

Wait 1-2 minutes for cameras to initialize, then check:

curl -s http://localhost:5000/api/stats | jq '.cameras | keys'  # Expect: list of camera names

---

Task 7: Deploy Farm Agent

Target: hf-pdocker-01 (192.168.86.192)

The existing farm-agent compose references 127.0.0.1:5050 (local registry, which is gone). We'll build the image directly from Gitea instead.

Step 1: Clone farm-manager repo and build agent image

cd /tmp
git clone http://192.168.86.100:3003/jeremy/farm-manager.git
cd farm-manager/agent
docker build -t farm-agent:latest .

Step 2: Update compose file to use local image

# Edit the compose file on NAS
sed -i 's|image: 127.0.0.1:5050/farm-agent:latest|image: farm-agent:latest|' /mnt/docker-data/compose/hf-pdocker-01/farm-agent/docker-compose.yml

Step 3: Deploy

cd /mnt/docker-data/compose/hf-pdocker-01/farm-agent
docker compose up -d

Step 4: Verify

docker ps --filter name=farm-agent --format "table {{.Names}}\t{{.Status}}"
curl -s http://localhost:8889/health  # Expect: healthy response

---

Task 8: Deploy Frigate-Notify v2

Target: hf-pdocker-01 (192.168.86.192)

Step 1: Clone frigate-notify repo

cd /tmp
git clone http://192.168.86.100:3003/jeremy/frigate-notify.git

Step 2: Build the image

cd /tmp/frigate-notify
docker build -t frigate-notify:latest .

Step 3: Create .env file with secrets

The compose file needs ANTHROPIC_API_KEY, MATRIX_ACCESS_TOKEN, and MATRIX_ROOM_ID. Retrieve these from Vaultwarden or use known values.

Known values from MEMORY.md:

• MATRIX_ACCESS_TOKEN: syt_ZnJpZ2F0ZQ_uaKUheedpkQBqtCaldsa_2ifSTC (bot: @frigate:hobbs.farm)
• MATRIX_ROOM_ID: !MTsFCQjnmLUNFGJePz:hobbs.farm
• ANTHROPIC_API_KEY: retrieve from vault or from bart's ai-agents .env

cat > /mnt/docker-data/compose/hf-pdocker-01/frigate-notify/.env << 'EOF'
ANTHROPIC_API_KEY=<retrieve from vault>
MATRIX_ACCESS_TOKEN=syt_ZnJpZ2F0ZQ_uaKUheedpkQBqtCaldsa_2ifSTC
MATRIX_ROOM_ID=!MTsFCQjnmLUNFGJePz:hobbs.farm
EOF

Step 4: Update compose to use built image instead of build context, and fix IPs

The compose file on NAS references build: context: . which won't work from NAS path. Update to use the locally built image. Also update MQTT and Matrix homeserver IPs since those services are on bart now.

Updated compose should have:

• image: frigate-notify:latest (instead of build:)
• FRIGATE_URL=http://localhost:5000 (Frigate is now local on same host)
• MQTT_HOST=192.168.86.167 (Mosquitto on bart)
• MATRIX_HOMESERVER=http://192.168.86.167:8008 (Synapse on bart)

Step 5: Deploy

cd /mnt/docker-data/compose/hf-pdocker-01/frigate-notify
docker compose up -d

Step 6: Verify

docker ps --filter name=frigate-notify --format "table {{.Names}}\t{{.Status}}"
curl -s http://localhost:5199/health  # Expect: healthy

---

Task 9: Restore Cron Jobs

Target: hf-pdocker-01 (192.168.86.192)

Step 1: Verify scripts are accessible on NAS

ls -la /mnt/docker-data/scripts/sync-frigate-to-nas.sh
ls -la /mnt/docker-data/scripts/cleanup-frigate-media.sh
cat /mnt/docker-data/scripts/sync-frigate-to-nas.sh
cat /mnt/docker-data/scripts/cleanup-frigate-media.sh

Step 2: Make scripts executable

chmod +x /mnt/docker-data/scripts/sync-frigate-to-nas.sh
chmod +x /mnt/docker-data/scripts/cleanup-frigate-media.sh

Step 3: Install crontab entries

(crontab -l 2>/dev/null; echo "0 2 * * * /mnt/docker-data/scripts/sync-frigate-to-nas.sh") | crontab -
(crontab -l 2>/dev/null; echo "0 3 * * 0 /mnt/docker-data/scripts/cleanup-frigate-media.sh") | crontab -

Step 4: Verify

crontab -l  # Expect: both cron entries listed

---

Task 10: Monitoring Integration

Target: hf-pdocker-01 + hf-pdocker-02 (Prometheus)

Step 1: Deploy Swarm global services (Telegraf + Node-exporter)

On hf-pdocker-01, deploy the monitoring stacks that run globally:

# Check if the stack files exist on NAS
ls /mnt/docker-data/compose/hf-pdocker-01/  # Look for telegraf, node-exporter stacks

If Telegraf and Node-exporter were Swarm stacks, redeploy them:

# These were global swarm services - redeploy the stacks
docker stack deploy -c /path/to/telegraf-stack.yml telegraf
docker stack deploy -c /path/to/node-exporter-stack.yml node-exporter

Step 2: Update Prometheus blackbox targets

On hf-pdocker-02 (192.168.86.100), ensure Prometheus has blackbox probe targets for the new services:

Targets to verify/add:

• http://192.168.86.192:5000 (Frigate)
• http://192.168.86.192:5199/health (Frigate-Notify)
• http://192.168.86.192:8889/health (Farm Agent)

Step 3: Hot-reload Prometheus

# On hf-pdocker-02:
curl -X POST http://localhost:9095/-/reload

Step 4: Verify in Grafana

Check Grafana at http://192.168.86.100:3001 — verify the new targets show as UP in the Prometheus targets page.

---

Phase 1 Complete Checklist

After all tasks, verify:

• Docker running on hf-pdocker-01
• NFS mount at /mnt/docker-data working
• 5TB disk mounted at /mnt/frigate-recordings
• Swarm cluster: 3 nodes (1 manager + 2 workers)
• Frigate running, cameras streaming, OpenVINO detector working
• Farm Agent responding on :8889
• Frigate-Notify v2 healthy on :5199
• Cron jobs installed for sync + cleanup
• Prometheus monitoring all new endpoints
• All blackbox probes green

Phase 2 (Future)

Migrate failover services back to hf-pdocker-01:

• Traefik (from hf-pdocker-02)
• Monitoring stack (from hf-pdocker-02)
• AdGuard Home (from hf-pdocker-02)
• Home Assistant, ESPHome, Mosquitto (from bart)
• Matrix/Synapse (from bart)
• Guacamole (from bart)
• AI Agents (from bart)
• Farm Manager API (new deploy)