hobbs-iac/docs/plans/2026-03-08-hf-pdocker-01-rebuild.md

# hf-pdocker-01 Rebuild — Phase 1 Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Rebuild hf-pdocker-01 from fresh Ubuntu 24.04 to a fully operational Swarm manager with Frigate NVR, Farm Agent, and Frigate-Notify v2.

**Architecture:** Fresh Ubuntu 24.04 on NVMe, Docker CE, NFS-backed storage from Synology NAS, 5TB USB disk for Frigate recordings. Docker Swarm re-initialized as single manager with hf-pdocker-02 and bart rejoined as workers. Services deployed via Compose files already on NAS.

**Tech Stack:** Ubuntu 24.04, Docker CE, Docker Swarm, NFS v3, Intel VA-API/OpenVINO, Python 3.12 (containers)

**Access:** SSH to hf-pdocker-01 (192.168.86.192), user `glyptic`, password `&Rip8a115`

**Important:** All tasks execute via SSH from the WSL environment. Use the `docker-swarm-ops` agent for all remote operations.

---

### Task 1: Install System Prerequisites

**Target:** hf-pdocker-01 (192.168.86.192)

**Step 1: Install base packages**

```bash
sudo apt-get update
sudo apt-get install -y nfs-common curl jq git rsync
```

**Step 2: Install Docker CE**

```bash
# Add Docker GPG key and repo
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```

**Step 3: Add glyptic to docker group**

```bash
sudo usermod -aG docker glyptic
```

**Step 4: Install Intel VA-API drivers for Frigate OpenVINO**

```bash
sudo apt-get install -y intel-media-va-driver-non-free vainfo
```

**Step 5: Verify**

```bash
docker --version         # Expect: Docker version 28.x
docker compose version   # Expect: Docker Compose version v2.x
vainfo                   # Expect: VA-API device listed (iHD driver, HD 630)
ls /dev/dri/             # Expect: card0, renderD128
```

---

### Task 2: Configure System Settings

**Target:** hf-pdocker-01 (192.168.86.192)

**Step 1: Set vm.swappiness to 10 (cluster standard)**

```bash
echo "vm.swappiness=10" | sudo tee -a /etc/sysctl.conf
sudo sysctl vm.swappiness=10
```

**Step 2: Verify**

```bash
sysctl vm.swappiness  # Expect: vm.swappiness = 10
```

---

### Task 3: Configure NFS Mounts

**Target:** hf-pdocker-01 (192.168.86.192)

**Step 1: Create mount points**

```bash
sudo mkdir -p /mnt/docker-data /mnt/infrastructure
```

**Step 2: Add NFS entries to fstab**

```bash
echo '192.168.86.30:/volume1/Docker /mnt/docker-data nfs rw,soft,timeo=300,intr,_netdev,nfsvers=3 0 0' | sudo tee -a /etc/fstab
echo '192.168.86.30:/volume1/Infrastructure /mnt/infrastructure nfs rw,hard,intr,_netdev 0 0' | sudo tee -a /etc/fstab
```

**Step 3: Mount and verify**

```bash
sudo mount -a
ls /mnt/docker-data/configs/    # Expect: frigate, frigate-notify, etc.
ls /mnt/infrastructure/margaret/  # Expect: openclaw.json, credentials/, etc.
```

---

### Task 4: Mount 5TB Frigate Recordings Disk

**Target:** hf-pdocker-01 (192.168.86.192)

**Step 1: Identify the disk and verify existing data**

```bash
sudo blkid /dev/sda1              # Check filesystem type (expect ext4)
sudo mkdir -p /mnt/frigate-recordings
sudo mount /dev/sda1 /mnt/frigate-recordings
ls /mnt/frigate-recordings/       # Expect: frigate recording directories
```

**Step 2: Add to fstab using UUID for reliability**

```bash
# Get UUID
UUID=$(sudo blkid -s UUID -o value /dev/sda1)
echo "UUID=$UUID /mnt/frigate-recordings ext4 defaults,_netdev 0 2" | sudo tee -a /etc/fstab
```

**Step 3: Verify**

```bash
df -h /mnt/frigate-recordings  # Expect: ~4.5T disk
```

---

### Task 5: Re-initialize Docker Swarm

**Target:** hf-pdocker-01 (192.168.86.192), then hf-pdocker-02 and bart

**Step 1: Initialize Swarm on hf-pdocker-01**

```bash
# On hf-pdocker-01:
docker swarm init --advertise-addr 192.168.86.192
```

Save the worker join token from the output.

**Step 2: Clean up stale Swarm state on workers**

```bash
# On hf-pdocker-02 (192.168.86.100):
docker swarm leave --force

# On bart (192.168.86.167):
docker swarm leave --force
```

**Step 3: Rejoin workers**

```bash
# On hf-pdocker-02:
docker swarm join --token <WORKER_TOKEN> 192.168.86.192:2377

# On bart:
docker swarm join --token <WORKER_TOKEN> 192.168.86.192:2377
```

**Step 4: Verify cluster**

```bash
# On hf-pdocker-01:
docker node ls   # Expect: 3 nodes, hf-pdocker-01 as Leader/Manager, others as Workers
```

**Step 5: Recreate overlay networks**

```bash
# On hf-pdocker-01:
docker network create --driver overlay --attachable traefik-public
```

---

### Task 6: Deploy Frigate NVR

**Target:** hf-pdocker-01 (192.168.86.192)

**Step 1: Update Frigate config MQTT address**

The Frigate config at `/mnt/docker-data/configs/frigate/config/config.yml` has MQTT pointed at `192.168.86.192:1883` (the old hf-pdocker-01 address). Mosquitto is currently running on bart (192.168.86.167) as a failover service.

```bash
# On hf-pdocker-01, edit the config to point MQTT to bart:
sed -i 's/host: 192.168.86.192/host: 192.168.86.167/' /mnt/docker-data/configs/frigate/config/config.yml
```

Verify the change:
```bash
grep -A2 "^mqtt:" /mnt/docker-data/configs/frigate/config/config.yml
# Expect: host: 192.168.86.167
```

**Step 2: Deploy Frigate**

```bash
cd /mnt/docker-data/compose/hf-pdocker-01/frigate
docker compose up -d
```

**Step 3: Verify**

```bash
docker ps --filter name=frigate --format "table {{.Names}}\t{{.Status}}"  # Expect: Up
curl -s http://localhost:5000/api/version  # Expect: Frigate version response
curl -s http://localhost:5000/api/stats | jq '.detectors'  # Expect: openvino detector ~10ms inference
```

Wait 1-2 minutes for cameras to initialize, then check:
```bash
curl -s http://localhost:5000/api/stats | jq '.cameras | keys'  # Expect: list of camera names
```

---

### Task 7: Deploy Farm Agent

**Target:** hf-pdocker-01 (192.168.86.192)

The existing farm-agent compose references `127.0.0.1:5050` (local registry, which is gone). We'll build the image directly from Gitea instead.

**Step 1: Clone farm-manager repo and build agent image**

```bash
cd /tmp
git clone http://192.168.86.100:3003/jeremy/farm-manager.git
cd farm-manager/agent
docker build -t farm-agent:latest .
```

**Step 2: Update compose file to use local image**

```bash
# Edit the compose file on NAS
sed -i 's|image: 127.0.0.1:5050/farm-agent:latest|image: farm-agent:latest|' /mnt/docker-data/compose/hf-pdocker-01/farm-agent/docker-compose.yml
```

**Step 3: Deploy**

```bash
cd /mnt/docker-data/compose/hf-pdocker-01/farm-agent
docker compose up -d
```

**Step 4: Verify**

```bash
docker ps --filter name=farm-agent --format "table {{.Names}}\t{{.Status}}"
curl -s http://localhost:8889/health  # Expect: healthy response
```

---

### Task 8: Deploy Frigate-Notify v2

**Target:** hf-pdocker-01 (192.168.86.192)

**Step 1: Clone frigate-notify repo**

```bash
cd /tmp
git clone http://192.168.86.100:3003/jeremy/frigate-notify.git
```

**Step 2: Build the image**

```bash
cd /tmp/frigate-notify
docker build -t frigate-notify:latest .
```

**Step 3: Create .env file with secrets**

The compose file needs `ANTHROPIC_API_KEY`, `MATRIX_ACCESS_TOKEN`, and `MATRIX_ROOM_ID`. Retrieve these from Vaultwarden or use known values.

Known values from MEMORY.md:
- `MATRIX_ACCESS_TOKEN`: `syt_ZnJpZ2F0ZQ_uaKUheedpkQBqtCaldsa_2ifSTC` (bot: @frigate:hobbs.farm)
- `MATRIX_ROOM_ID`: `!MTsFCQjnmLUNFGJePz:hobbs.farm`
- `ANTHROPIC_API_KEY`: retrieve from vault or from bart's ai-agents .env

```bash
cat > /mnt/docker-data/compose/hf-pdocker-01/frigate-notify/.env << 'EOF'
ANTHROPIC_API_KEY=<retrieve from vault>
MATRIX_ACCESS_TOKEN=syt_ZnJpZ2F0ZQ_uaKUheedpkQBqtCaldsa_2ifSTC
MATRIX_ROOM_ID=!MTsFCQjnmLUNFGJePz:hobbs.farm
EOF
```

**Step 4: Update compose to use built image instead of build context, and fix IPs**

The compose file on NAS references `build: context: .` which won't work from NAS path. Update to use the locally built image. Also update MQTT and Matrix homeserver IPs since those services are on bart now.

Updated compose should have:
- `image: frigate-notify:latest` (instead of `build:`)
- `FRIGATE_URL=http://localhost:5000` (Frigate is now local on same host)
- `MQTT_HOST=192.168.86.167` (Mosquitto on bart)
- `MATRIX_HOMESERVER=http://192.168.86.167:8008` (Synapse on bart)

**Step 5: Deploy**

```bash
cd /mnt/docker-data/compose/hf-pdocker-01/frigate-notify
docker compose up -d
```

**Step 6: Verify**

```bash
docker ps --filter name=frigate-notify --format "table {{.Names}}\t{{.Status}}"
curl -s http://localhost:5199/health  # Expect: healthy
```

---

### Task 9: Restore Cron Jobs

**Target:** hf-pdocker-01 (192.168.86.192)

**Step 1: Verify scripts are accessible on NAS**

```bash
ls -la /mnt/docker-data/scripts/sync-frigate-to-nas.sh
ls -la /mnt/docker-data/scripts/cleanup-frigate-media.sh
cat /mnt/docker-data/scripts/sync-frigate-to-nas.sh
cat /mnt/docker-data/scripts/cleanup-frigate-media.sh
```

**Step 2: Make scripts executable**

```bash
chmod +x /mnt/docker-data/scripts/sync-frigate-to-nas.sh
chmod +x /mnt/docker-data/scripts/cleanup-frigate-media.sh
```

**Step 3: Install crontab entries**

```bash
(crontab -l 2>/dev/null; echo "0 2 * * * /mnt/docker-data/scripts/sync-frigate-to-nas.sh") | crontab -
(crontab -l 2>/dev/null; echo "0 3 * * 0 /mnt/docker-data/scripts/cleanup-frigate-media.sh") | crontab -
```

**Step 4: Verify**

```bash
crontab -l  # Expect: both cron entries listed
```

---

### Task 10: Monitoring Integration

**Target:** hf-pdocker-01 + hf-pdocker-02 (Prometheus)

**Step 1: Deploy Swarm global services (Telegraf + Node-exporter)**

On hf-pdocker-01, deploy the monitoring stacks that run globally:

```bash
# Check if the stack files exist on NAS
ls /mnt/docker-data/compose/hf-pdocker-01/  # Look for telegraf, node-exporter stacks
```

If Telegraf and Node-exporter were Swarm stacks, redeploy them:
```bash
# These were global swarm services - redeploy the stacks
docker stack deploy -c /path/to/telegraf-stack.yml telegraf
docker stack deploy -c /path/to/node-exporter-stack.yml node-exporter
```

**Step 2: Update Prometheus blackbox targets**

On hf-pdocker-02 (192.168.86.100), ensure Prometheus has blackbox probe targets for the new services:

Targets to verify/add:
- `http://192.168.86.192:5000` (Frigate)
- `http://192.168.86.192:5199/health` (Frigate-Notify)
- `http://192.168.86.192:8889/health` (Farm Agent)

**Step 3: Hot-reload Prometheus**

```bash
# On hf-pdocker-02:
curl -X POST http://localhost:9095/-/reload
```

**Step 4: Verify in Grafana**

Check Grafana at http://192.168.86.100:3001 — verify the new targets show as UP in the Prometheus targets page.

---

## Phase 1 Complete Checklist

After all tasks, verify:
- [ ] Docker running on hf-pdocker-01
- [ ] NFS mount at /mnt/docker-data working
- [ ] 5TB disk mounted at /mnt/frigate-recordings
- [ ] Swarm cluster: 3 nodes (1 manager + 2 workers)
- [ ] Frigate running, cameras streaming, OpenVINO detector working
- [ ] Farm Agent responding on :8889
- [ ] Frigate-Notify v2 healthy on :5199
- [ ] Cron jobs installed for sync + cleanup
- [ ] Prometheus monitoring all new endpoints
- [ ] All blackbox probes green

## Phase 2 (Future)

Migrate failover services back to hf-pdocker-01:
- Traefik (from hf-pdocker-02)
- Monitoring stack (from hf-pdocker-02)
- AdGuard Home (from hf-pdocker-02)
- Home Assistant, ESPHome, Mosquitto (from bart)
- Matrix/Synapse (from bart)
- Guacamole (from bart)
- AI Agents (from bart)
- Farm Manager API (new deploy)