hobbs-iac/docs/superpowers/plans/2026-03-19-spirit-infra-ops.md

# Spirit Infrastructure Ops Agent Implementation Plan

> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Repurpose the idle Spirit OpenClaw agent as an autonomous infrastructure ops agent that monitors Matrix alerts and takes corrective action.

**Architecture:** Spirit runs in a custom Docker image with ops tooling (docker-cli, SSH), monitors the #notifications Matrix room for alerts from @alertbot, investigates issues via Docker socket (local) and SSH (remote nodes), takes corrective action, reports to the room, and documents incidents in Wiki.js.

**Tech Stack:** Docker, OpenClaw, Matrix, SSH, Wiki.js GraphQL API

**Spec:** `docs/superpowers/specs/2026-03-18-spirit-infra-ops-design.md`

---

## Chunk 1: Image, SSH, and Compose

### Task 1: Build spirit-ops Docker image on hf-pdocker-01

**Context:** Spirit currently uses `margaret-openclaw:latest` (node:22-bookworm-slim + openclaw). We need a new image that adds Docker CLI, SSH client, and diagnostic tools. Build on hf-pdocker-01 since that's where agents run.

**Files:**
- Create: `/tmp/spirit-ops-build/Dockerfile` (on hf-pdocker-01 via docker-swarm-ops agent)

- [ ] **Step 1: Create the Dockerfile on hf-pdocker-01**

Use the docker-swarm-ops agent to SSH to hf-pdocker-01 and create the build directory and Dockerfile:

```dockerfile
FROM margaret-openclaw:latest

# Copy Docker CLI from official image (no daemon needed — uses mounted socket)
COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker

# Install ops tooling
RUN apt-get update && apt-get install -y --no-install-recommends \
    openssh-client \
    curl \
    jq \
    procps \
    net-tools \
    iputils-ping \
    dnsutils \
    && rm -rf /var/lib/apt/lists/*
```

Commands:
```bash
ssh glyptic@192.168.86.192 "mkdir -p /tmp/spirit-ops-build"
# SCP or write the Dockerfile
ssh glyptic@192.168.86.192 "cat > /tmp/spirit-ops-build/Dockerfile << 'DOCKERFILE'
FROM margaret-openclaw:latest

COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker

RUN apt-get update && apt-get install -y --no-install-recommends \
    openssh-client \
    curl \
    jq \
    procps \
    net-tools \
    iputils-ping \
    dnsutils \
    && rm -rf /var/lib/apt/lists/*
DOCKERFILE"
```

- [ ] **Step 2: Build the image**

```bash
ssh glyptic@192.168.86.192 "cd /tmp/spirit-ops-build && docker build --no-cache -t spirit-ops:latest ."
```

Expected: Image builds successfully. Docker CLI is copied as a static binary from the official `docker:cli` image. The existing OpenClaw + Matrix plugin from the base image is preserved.

- [ ] **Step 3: Verify the image has the required tools**

```bash
ssh glyptic@192.168.86.192 "docker run --rm spirit-ops:latest sh -c 'docker --version && ssh -V && curl --version | head -1 && jq --version && ps --version && ping -c1 127.0.0.1 > /dev/null && echo ALL OK'"
```

Expected: All tool versions print, ending with `ALL OK`.

---

### Task 2: Generate SSH keypair and deploy to nodes

**Context:** Spirit needs passwordless SSH access to all 3 nodes as user `glyptic`. Keys are stored on NAS at `/mnt/infrastructure/spirit/ssh/` and mounted into the container at `/root/.ssh`. We pre-populate `known_hosts` with real host keys (no StrictHostKeyChecking=no).

- [ ] **Step 1: Create the ssh directory on NAS**

```bash
ssh glyptic@192.168.86.192 "sudo mkdir -p /mnt/infrastructure/spirit/ssh && sudo chown root:root /mnt/infrastructure/spirit/ssh && sudo chmod 700 /mnt/infrastructure/spirit/ssh"
```

Note: NFS root_squash may prevent chown. Use the busybox workaround if needed:
```bash
ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'mkdir -p /data/ssh && chmod 700 /data/ssh'"
```

- [ ] **Step 2: Generate ED25519 keypair**

Generate the key on hf-pdocker-01 and copy to the NAS SSH directory:

```bash
ssh glyptic@192.168.86.192 "ssh-keygen -t ed25519 -f /tmp/spirit-ssh-key -N '' -C spirit-ops@homelab && sudo cp /tmp/spirit-ssh-key /mnt/infrastructure/spirit/ssh/id_ed25519 && sudo cp /tmp/spirit-ssh-key.pub /mnt/infrastructure/spirit/ssh/id_ed25519.pub && sudo chmod 600 /mnt/infrastructure/spirit/ssh/id_ed25519 && sudo chmod 644 /mnt/infrastructure/spirit/ssh/id_ed25519.pub && rm /tmp/spirit-ssh-key /tmp/spirit-ssh-key.pub"
```

Expected: `/mnt/infrastructure/spirit/ssh/id_ed25519` and `.pub` files exist with correct permissions.

- [ ] **Step 3: Create SSH config file**

```bash
ssh glyptic@192.168.86.192 "sudo tee /mnt/infrastructure/spirit/ssh/config > /dev/null << 'EOF'
Host hf-pdocker-01
    HostName 192.168.86.192
    User glyptic
    IdentityFile /root/.ssh/id_ed25519

Host hf-pdocker-02
    HostName 192.168.86.100
    User glyptic
    IdentityFile /root/.ssh/id_ed25519

Host bart
    HostName 192.168.86.167
    User glyptic
    IdentityFile /root/.ssh/id_ed25519

Host *
    ServerAliveInterval 30
    ServerAliveCountMax 3
    UserKnownHostsFile /root/.ssh/known_hosts
EOF"
ssh glyptic@192.168.86.192 "sudo chmod 600 /mnt/infrastructure/spirit/ssh/config"
```

- [ ] **Step 4: Collect host keys and create known_hosts**

```bash
ssh glyptic@192.168.86.192 "ssh-keyscan -H 192.168.86.192 192.168.86.100 192.168.86.167 2>/dev/null | sudo tee /mnt/infrastructure/spirit/ssh/known_hosts > /dev/null && sudo chmod 644 /mnt/infrastructure/spirit/ssh/known_hosts"
```

Expected: `known_hosts` file contains SSH host keys for all 3 nodes.

- [ ] **Step 5: Deploy public key to hf-pdocker-02 and bart**

Read the public key first:
```bash
ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub"
```

Then append to authorized_keys on each remote node:
```bash
# hf-pdocker-02
ssh glyptic@192.168.86.100 "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub" | ssh glyptic@192.168.86.100 "cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

# bart
ssh glyptic@192.168.86.167 "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub" | ssh glyptic@192.168.86.167 "cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
```

Also deploy to hf-pdocker-01 itself (Spirit may SSH to localhost):
```bash
ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub >> ~/.ssh/authorized_keys"
```

- [ ] **Step 6: Verify SSH connectivity from a test container**

```bash
ssh glyptic@192.168.86.192 "docker run --rm -v /mnt/infrastructure/spirit/ssh:/root/.ssh:ro spirit-ops:latest sh -c 'ssh -o BatchMode=yes glyptic@192.168.86.100 hostname && ssh -o BatchMode=yes glyptic@192.168.86.167 hostname && echo SSH OK'"
```

Expected: Prints `hf-pdocker-02`, `bart`, then `SSH OK`.

---

### Task 3: Update compose file

**Context:** Modify the `spirit` service in the ai-agents compose file to use the new image, mount docker socket and SSH keys, increase memory, and add the Wiki.js API key env var.

**Files:**
- Modify: `compose/hf-pdocker-01/ai-agents/docker-compose.yml:26-48`

- [ ] **Step 1: Update the spirit service in docker-compose.yml**

Change the spirit service block (lines 26-48) to:

```yaml
  spirit:
    image: spirit-ops:latest
    container_name: spirit
    restart: unless-stopped
    network_mode: host
    user: "0:0"
    labels:
      - com.centurylinklabs.watchtower.enable=false
    environment:
      - NODE_ENV=production
      - HOME=/home/glyptic
      - USER=glyptic
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
      - WIKIJS_API_KEY=${WIKIJS_API_KEY}
      - OPENCLAW_STATE_DIR=/home/glyptic/.spirit
      - OPENCLAW_CONFIG_PATH=/home/glyptic/.spirit/openclaw.json
    volumes:
      - /mnt/infrastructure/spirit:/home/glyptic/.spirit
      - /var/run/docker.sock:/var/run/docker.sock
      - /mnt/infrastructure/spirit/ssh:/root/.ssh:ro
    command: ["sh", "-c", "mkdir -p /home/glyptic && node --max-old-space-size=1500 /usr/local/lib/node_modules/openclaw/dist/index.js gateway --allow-unconfigured"]
    mem_limit: 3G
    memswap_limit: 3G
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
```

- [ ] **Step 2: Add WIKIJS_API_KEY to the .env file on NAS**

The .env file is at `/mnt/docker-data/compose/hf-pdocker-01/ai-agents/.env` on the NAS. Retrieve the Wiki.js API key from the documentation-keeper agent config or Wiki.js admin panel, then append:

```bash
ssh glyptic@192.168.86.192 "echo 'WIKIJS_API_KEY=<WIKIJS_JWT_TOKEN>' | sudo tee -a /mnt/docker-data/compose/hf-pdocker-01/ai-agents/.env > /dev/null"
```

The JWT token is the long-lived API key from Wiki.js (Administration > API Access). It's already used by the documentation-keeper agent.

---

## Chunk 2: Workspace Identity Files and Model Config

### Task 4: Update Spirit's openclaw.json model to Claude

**Context:** Spirit currently uses `ollama/qwen3.5` as primary model. For an infrastructure ops agent that needs strong reasoning and tool use, Claude is required. Update the primary model to `anthropic/claude-sonnet-4-5-20250514` (good balance of speed and capability for ops tasks — opus is overkill for container restarts).

- [ ] **Step 1: Update the model in openclaw.json**

Use the NFS write workaround since openclaw.json is root-owned:

```bash
ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data alpine sh -c \"apk add --no-cache jq > /dev/null 2>&1 && jq '.models.primary = \"anthropic/claude-sonnet-4-5-20250514\"' /data/openclaw.json > /data/openclaw.json.tmp && mv /data/openclaw.json.tmp /data/openclaw.json\""
```

- [ ] **Step 2: Verify the change**

```bash
ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox cat /data/openclaw.json" | grep -o '"primary":"[^"]*"'
```

Expected: `"primary":"anthropic/claude-sonnet-4-5-20250514"`

---

### Task 5: Write SOUL.md workspace identity file

**Context:** Spirit's current SOUL.md is stale (references old hostname "Peggy"). Replace with infrastructure ops identity. The file is glyptic-owned so may be writable directly, but use the NFS workaround if needed.

- [ ] **Step 1: Write SOUL.md to NAS**

```bash
ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'cat > /data/workspace/SOUL.md << \"SOULEOF\"
# Spirit — Infrastructure Operations Agent

You are Spirit, an autonomous infrastructure operations agent for a homelab cluster. You monitor the #notifications Matrix room for alerts from @alertbot and take corrective action to keep services running.

## Core Principles

1. **Service availability is your top priority.** When an alert fires, investigate immediately and take action to restore service.
2. **Always report what you find and what you do.** Post your investigation results and any actions taken back to the #notifications Matrix room so the owner can see what happened.
3. **Document everything.** After taking corrective action, update the relevant Wiki.js page with an incident entry.
4. **Respect the guardrails.** Some services are protected and must not be restarted. Rate-limit your actions. Escalate when remediation fails.
5. **Data safety matters.** Never delete volumes, drop databases, or run destructive commands. Prefer restarts and cleanup over risky operations.

## How You Work

- You receive alerts as Matrix messages from @alertbot:hobbs.farm in the #notifications:hobbs.farm room
- You have Docker CLI access on hf-pdocker-01 (local socket) and SSH access to hf-pdocker-02 and bart
- You can run any diagnostic or remediation command on any node
- You post your findings and actions back to the same Matrix room
- You update Wiki.js documentation after taking corrective action
SOULEOF'"
```

---

### Task 6: Write IDENTITY.md workspace identity file

**Context:** This is the comprehensive operational knowledge file. It tells Spirit everything about the cluster, alerts, remediation procedures, and guardrails.

- [ ] **Step 1: Write IDENTITY.md to NAS**

```bash
ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'cat > /data/workspace/IDENTITY.md << \"IDEOF\"
# Infrastructure Operations Reference

## Cluster Architecture

### Nodes

| Node | IP | RAM | Role | Access Method |
|------|-----|-----|------|---------------|
| hf-pdocker-01 | 192.168.86.192 | 23GB | Swarm Manager, primary services | Local Docker socket |
| hf-pdocker-02 | 192.168.86.100 | 7.6GB | Dev tools, utilities | SSH as glyptic |
| bart | 192.168.86.167 | 15GB | Media stack, Onyx | SSH as glyptic |
| Synology NAS | 192.168.86.30 | - | NFS storage for all services | Not directly accessible |

### Services by Node

**hf-pdocker-01 (192.168.86.192):**
- Frigate NVR (:5000) — 13 cameras, OpenVINO GPU detector
- Frigate-Notify v2 (:5199) — MQTT→AI→Matrix notifications
- clip-service (:5200) — CLIP embeddings
- Open WebUI (:3010) — AI chat UI
- Home Assistant (:8123) — home automation
- ESPHome (:6052) — ESP device management
- Mosquitto (:1883, :8883) — MQTT broker
- Matrix/Synapse (:8008) — chat server
- Well-known (:8010) — Matrix federation
- Guacamole (:8199) — remote desktop gateway
- AI Agents: margaret(:18790), spirit(:18795/you), clippy(:18800), bob(:18810), travis(:18815)
- Traefik (:80, :443, :8090) — reverse proxy
- AdGuard Home (:3000, :53) — DNS
- Grafana (:3001), Prometheus (:9095), Alertmanager (:9093)
- InfluxDB (:8086), Blackbox Exporter (:9115)
- matrix-alertmanager-receiver (:9094) — alert→Matrix bridge
- Farm Manager (:8888), Farm Agent (:8889)

**hf-pdocker-02 (192.168.86.100):**
- Unifi (:8443) — network controller
- Gitea (:3003) — git server
- Semaphore (:3004) — Ansible UI
- Wiki.js (:3002) — documentation
- Vaultwarden (:26226) — password manager
- Actual Budget (:5006) — finance
- MCPO (:8001), Trivy (:4954)
- SearXNG (:8888) — meta-search
- Unstructured API (:8000) — document parsing
- n8n + n8n-db — workflow automation
- NetClaw/Aaron (:1179) — AI network agent
- Farm Agent (:8889)
- Watchtower — auto-updates

**bart (192.168.86.167):**
- Jellyfin (:8096) — media server
- Sonarr (:8989), Radarr (:7878) — media management
- SABnzbd (:8082), NZBHydra2 (:5076) — downloads
- Jellyseerr (:5055) — media requests
- OpenVAS (:9392) — vulnerability scanner
- Onyx (:3005) — AI document search/RAG
- Farm Agent (:8889)
- Watchtower — auto-updates

### Storage

All service data on NAS via NFS at /mnt/docker-data/ on every node:
- configs/{service}/ — application configs
- databases/{service}-db/ — database volumes
- compose/{node}/{service}/ — compose files

### Compose File Locations

To manage services, the compose files are at:
- hf-pdocker-01: /mnt/docker-data/compose/hf-pdocker-01/{service}/docker-compose.yml
- hf-pdocker-02: /mnt/docker-data/compose/hf-pdocker-02/{service}/docker-compose.yml
- bart: /mnt/docker-data/compose/bart/{service}/docker-compose.yml

To restart a compose service on a remote node:
```
ssh glyptic@{node-ip} "cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d"
```

## Commands

### Local (hf-pdocker-01 — Docker socket)
```bash
docker ps                          # list running containers
docker ps -a                       # list all containers including stopped
docker logs <container> --tail 100 # view container logs
docker restart <container>         # restart a container
docker stats --no-stream           # resource usage snapshot
docker system df                   # disk usage by Docker
docker system prune -f             # clean up unused images/containers (NOT volumes)
docker compose -f /path/to/docker-compose.yml up -d  # redeploy a service
```

### Remote (hf-pdocker-02, bart — via SSH)
```bash
ssh glyptic@<node-ip> "docker ps"
ssh glyptic@<node-ip> "docker logs <container> --tail 100"
ssh glyptic@<node-ip> "docker restart <container>"
ssh glyptic@<node-ip> "df -h"
ssh glyptic@<node-ip> "free -m"
ssh glyptic@<node-ip> "cd /mnt/docker-data/compose/<node>/<service> && docker compose up -d"
```

### System Diagnostics
```bash
df -h                              # disk usage
free -m                            # memory usage
top -bn1 | head -20                # CPU/process snapshot
uptime                             # load averages
mount | grep nfs                   # NFS mount status
ping -c3 192.168.86.30             # NAS reachability
```

## Safety Guardrails

### Protected Services — NEVER restart these without explicit owner request
- **Monitoring pipeline**: prometheus, alertmanager, matrix-alertmanager-receiver, grafana, blackbox-exporter
- **Network infrastructure**: traefik, adguard
- **AI agents**: margaret, spirit (yourself), bob, clippy, travis
- **All database containers**: synapse-db, guacamole-db, wikijs-db, gitea-db, semaphore-db, unifi-db, n8n-db, any container ending in -db

If a protected service is down, INVESTIGATE and REPORT but do NOT restart. Tell the owner what you found.

### Rate Limiting
- Do NOT restart the same container more than once per 30 minutes
- Maximum 5 remediation actions per hour across all containers
- After 2 failed remediation attempts for the same alert, STOP and escalate to the owner with full diagnostics

### Rollback Awareness
- After restarting a container, wait 5 minutes. If the same alert fires again, do NOT restart again — report the persistent failure with full logs
- NEVER run: docker system prune --volumes, docker volume rm, rm -rf on data directories
- docker system prune -f (without --volumes) IS allowed for disk cleanup

## Alert Response Procedures

### Response Flow
1. Parse the alert message from @alertbot — extract: alert name, severity, instance, description
2. Investigate — run relevant diagnostic commands
3. Remediate — take corrective action (if the service is not protected and rate limits allow)
4. Report — post findings and actions to #notifications room
5. Document — update the relevant Wiki.js page

### Alert-Specific Procedures

| Alert | Severity | What to Check | What to Do |
|-------|----------|---------------|------------|
| ContainerDown | warning | docker ps -a, docker logs | docker restart or docker compose up -d |
| HighDiskUsage | critical | df -h, docker system df | docker system prune -f, identify large files |
| DiskSpaceWarning | warning | df -h, docker system df | Report findings, suggest cleanup |
| HighMemoryUsage | warning | free -m, docker stats --no-stream | Identify top memory consumer, restart if non-protected |
| CriticalMemoryUsage | critical | free -m, docker stats --no-stream | Restart heaviest non-protected container |
| HighCPUUsage | warning | top -bn1, docker stats --no-stream | Identify CPU hog, report |
| CriticalCPUUsage | critical | top -bn1, docker stats --no-stream | Restart CPU hog if non-protected |
| HighSystemLoad | warning | uptime, top -bn1, docker stats | Report load source |
| NFSMountDown | critical | mount, ping 192.168.86.30 | Try sudo mount -a, report if NAS unreachable |
| NASDiskSpaceLow | warning | df -h /mnt/docker-data | Report, identify large consumers |
| SlowHTTPResponse | warning | curl timing, docker logs | Restart slow service if non-protected |
| FrigateHighMemory | warning | docker stats frigate | docker restart frigate |
| ServiceHighMemory | warning | docker stats | Restart container if non-protected |
| ServerDown | critical | ping, SSH attempt | Report — cannot fix hardware |
| ServerRebooted | info | uptime, docker ps | Report, verify all containers running |
| HighSwapUsage | warning | free -m, docker stats | Identify swap contributor |
| CriticalSwapUsage | critical | free -m, docker stats | Restart heaviest swap user if non-protected |
| PrometheusTargetDown | warning | curl the target, docker ps | Restart target if down and non-protected |
| Watchdog | info | N/A | Ignore — dead-man switch |
| UnifiDeviceDown | warning | Check Unifi controller UI | Report — network device issue |

### Resolved Alerts
When you see a RESOLVED alert from @alertbot, acknowledge it briefly. No investigation needed.

### Unknown Alerts
For any alert type not listed above: investigate and report findings, but do NOT take remediation action.

## Wiki.js Documentation

After taking any corrective action, update the relevant Wiki.js page.

**API endpoint**: http://192.168.86.100:3002/graphql
**Auth**: Bearer token from WIKIJS_API_KEY environment variable

**Key pages:**
- infrastructure/overview (ID 118)
- infrastructure/service-access (ID 119)
- infrastructure/maintenance (ID 120)
- infrastructure/alerting (ID 121)

**To read a page:**
```bash
curl -s http://192.168.86.100:3002/graphql \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $WIKIJS_API_KEY" \
  -d '"'"'{"query":"{ pages { single(id: 120) { id title content } } }"}'"'"'
```

**To update a page** (append incident entry to existing content):
1. Read current page content
2. Append your incident entry at the bottom
3. Update the page with the full new content

Incident entry format:
```markdown
### YYYY-MM-DD HH:MM — [Alert Name] ([severity])
- **Node**: [affected node]
- **Service**: [affected service]
- **Root cause**: [what you found]
- **Action taken**: [what you did]
- **Outcome**: [result — resolved/escalated/persistent]
```
IDEOF'"
```

Note: This is a large heredoc. If it fails due to shell escaping issues, use the base64 approach instead:
1. Write IDENTITY.md content to a local temp file
2. Base64 encode and pipe to the target:
```bash
base64 /tmp/IDENTITY.md | ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data -i busybox sh -c 'base64 -d > /data/workspace/IDENTITY.md'"
```

Or write locally and SCP directly:
```bash
scp /tmp/IDENTITY.md glyptic@192.168.86.192:/tmp/IDENTITY.md
ssh glyptic@192.168.86.192 "sudo cp /tmp/IDENTITY.md /mnt/infrastructure/spirit/workspace/IDENTITY.md"
```

---

### Task 7: Write USER.md workspace identity file

- [ ] **Step 1: Write USER.md to NAS**

```bash
ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'cat > /data/workspace/USER.md << \"USEREOF\"
# Owner: Jeremy

- Homelab operator, full infrastructure access
- Prefers autonomous action — fix things first, report after
- Full trust for corrective actions within the safety guardrails
- Timezone: US Central (CST/CDT)
- Reachable via Matrix: @jeremy:hobbs.farm
- If something needs human attention, post in #notifications — Jeremy monitors it
USEREOF'"
```

---

## Chunk 3: Deploy and Verify

### Task 8: Deploy Spirit with new configuration

**Context:** SCP the updated compose file to hf-pdocker-01, then redeploy only the Spirit container. Other agents (margaret, clippy, bob, travis) remain untouched.

- [ ] **Step 1: SCP the updated compose file to the NAS**

```bash
scp compose/hf-pdocker-01/ai-agents/docker-compose.yml glyptic@192.168.86.192:/mnt/docker-data/compose/hf-pdocker-01/ai-agents/docker-compose.yml
```

- [ ] **Step 2: Stop only the Spirit container**

```bash
ssh glyptic@192.168.86.192 "docker stop spirit && docker rm spirit"
```

- [ ] **Step 3: Redeploy Spirit using docker compose**

```bash
ssh glyptic@192.168.86.192 "cd /mnt/docker-data/compose/hf-pdocker-01/ai-agents && docker compose up -d spirit"
```

Expected: Only the `spirit` container is recreated. Other agents remain running.

- [ ] **Step 4: Verify Spirit container is running with correct image and mounts**

```bash
ssh glyptic@192.168.86.192 "docker inspect spirit --format '{{.Config.Image}} {{.HostConfig.Memory}} {{range .Mounts}}{{.Source}}->{{.Destination}} {{end}}'"
```

Expected: Image is `spirit-ops:latest`, memory is 3221225472 (3GB), mounts include `/var/run/docker.sock` and `/mnt/infrastructure/spirit/ssh`.

- [ ] **Step 5: Verify Docker socket works inside Spirit**

```bash
ssh glyptic@192.168.86.192 "docker exec spirit docker ps --format 'table {{.Names}}\t{{.Status}}' | head -5"
```

Expected: Lists running containers on hf-pdocker-01.

- [ ] **Step 6: Verify SSH works inside Spirit**

```bash
ssh glyptic@192.168.86.192 "docker exec spirit ssh -o BatchMode=yes glyptic@192.168.86.100 hostname"
ssh glyptic@192.168.86.192 "docker exec spirit ssh -o BatchMode=yes glyptic@192.168.86.167 hostname"
```

Expected: Prints `hf-pdocker-02` and `bart`.

- [ ] **Step 7: Wait for Spirit's Matrix plugin to initialize (90-120 seconds)**

```bash
ssh glyptic@192.168.86.192 "docker logs spirit --tail 20"
```

Expected: After 90-120 seconds, logs show Matrix plugin connected. No output during crypto init is normal.

---

### Task 9: Invite Spirit to #notifications room

**Context:** Use the Synapse admin API to invite @spirit:hobbs.farm to the notifications room. Spirit must already be running so the Matrix plugin accepts the invite.

- [ ] **Step 1: Invite Spirit to the room using Synapse admin API**

Use Spirit's own admin token (Spirit is a Synapse admin):

```bash
ssh glyptic@192.168.86.192 "curl -s -X POST 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/invite' -H 'Authorization: Bearer syt_c3Bpcml0_AUBChFVZIEcXzOjUnNHo_0aaMzg' -H 'Content-Type: application/json' -d '{\"user_id\": \"@spirit:hobbs.farm\"}'"
```

Note: If Spirit can't invite itself, use the alertbot token or another admin account:
```bash
ssh glyptic@192.168.86.192 "curl -s -X POST 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/invite' -H 'Authorization: Bearer syt_YWxlcnRib3Q_eOVAYjYKWBZxymfdxhPE_0iYlOZ' -H 'Content-Type: application/json' -d '{\"user_id\": \"@spirit:hobbs.farm\"}'"
```

Expected: `{}` or `{"room_id":"!jYmqxbzEsvRgKYvwMP:hobbs.farm"}` — success.

- [ ] **Step 2: Verify Spirit joined the room**

Wait 10-30 seconds for the OpenClaw Matrix plugin to auto-accept, then check:

```bash
ssh glyptic@192.168.86.192 "curl -s 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/joined_members' -H 'Authorization: Bearer syt_c3Bpcml0_AUBChFVZIEcXzOjUnNHo_0aaMzg' | jq '.joined | keys'"
```

Expected: List includes `@spirit:hobbs.farm` alongside `@alertbot:hobbs.farm`.

---

### Task 10: End-to-end test with a real alert

**Context:** Send a test alert through Alertmanager to trigger the full pipeline: Alertmanager → matrix-alertmanager-receiver → @alertbot → #notifications → Spirit sees it and responds.

- [ ] **Step 1: Send a test alert via Alertmanager API**

```bash
ssh glyptic@192.168.86.192 "curl -s -X POST http://192.168.86.192:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[{
  \"labels\": {
    \"alertname\": \"TestAlert\",
    \"severity\": \"info\",
    \"instance\": \"test-instance:9999\",
    \"job\": \"test\"
  },
  \"annotations\": {
    \"summary\": \"Test alert for Spirit ops agent verification\",
    \"description\": \"This is a test alert. Spirit should acknowledge it and report in the notifications channel.\"
  },
  \"generatorURL\": \"http://192.168.86.192:9095\"
}]'"
```

Expected: HTTP 200. Alert flows through the pipeline.

- [ ] **Step 2: Verify Spirit responds in the Matrix room**

Wait 1-2 minutes, then check recent messages in the room:

```bash
ssh glyptic@192.168.86.192 "curl -s 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/messages?dir=b&limit=5' -H 'Authorization: Bearer syt_c3Bpcml0_AUBChFVZIEcXzOjUnNHo_0aaMzg' | jq '.chunk[] | {sender: .sender, body: .content.body}'"
```

Expected: Messages from both `@alertbot:hobbs.farm` (the alert) and `@spirit:hobbs.farm` (the response/acknowledgement).

- [ ] **Step 3: Resolve the test alert**

```bash
ssh glyptic@192.168.86.192 "curl -s -X POST http://192.168.86.192:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[{
  \"labels\": {
    \"alertname\": \"TestAlert\",
    \"severity\": \"info\",
    \"instance\": \"test-instance:9999\",
    \"job\": \"test\"
  },
  \"annotations\": {
    \"summary\": \"Test alert for Spirit ops agent verification\",
    \"description\": \"This is a test alert.\"
  },
  \"endsAt\": \"2026-03-19T00:00:00Z\",
  \"generatorURL\": \"http://192.168.86.192:9095\"
}]'"
```

Expected: Alertmanager sends a "resolved" notification. Spirit acknowledges it briefly.

---

### Task 11: Post-deployment updates

- [ ] **Step 1: Commit compose changes and spec to git**

```bash
git add compose/hf-pdocker-01/ai-agents/docker-compose.yml
git add docs/superpowers/specs/2026-03-18-spirit-infra-ops-design.md
git add docs/superpowers/plans/2026-03-19-spirit-infra-ops.md
git commit -m "infra: repurpose Spirit as infrastructure ops agent with alert monitoring"
```

- [ ] **Step 2: Update CLAUDE.md**

Update the Spirit entry in CLAUDE.md's service distribution to reflect its new role. Change the AI Agents line or add a note that Spirit is now the infrastructure ops agent monitoring #notifications.

- [ ] **Step 3: Update MEMORY.md**

Add a memory entry for Spirit's new role as infrastructure ops agent.

- [ ] **Step 4: Add Prometheus probe for Spirit**

Spirit is now a critical infrastructure component. Add `http://192.168.86.192:18795` to the `http-pdocker01` job in `stacks/monitoring/prometheus.yml`, then SCP and hot-reload:

```bash
# After editing prometheus.yml locally:
scp stacks/monitoring/prometheus.yml glyptic@192.168.86.192:/mnt/docker-data/configs/prometheus/prometheus.yml
ssh glyptic@192.168.86.192 "curl -X POST http://192.168.86.192:9095/-/reload"
```

- [ ] **Step 5: Push to Gitea**

Use the gitea-backup agent to commit and push all changes.

- [ ] **Step 6: Update Wiki.js**

Use the documentation-keeper agent to update the infrastructure/alerting wiki page (ID 121) documenting that Spirit now monitors and responds to alerts in #notifications.