infra: repurpose Spirit as autonomous infrastructure ops agent

Spirit now monitors #notifications Matrix room for Prometheus alerts, auto-investigates issues, takes corrective action (container restarts, disk cleanup, etc.), and documents incidents in Wiki.js. Changes: - Custom spirit-ops Docker image (Docker CLI + SSH + diagnostic tools) - Docker socket mount for local container management - SSH keys for remote access to hf-pdocker-02 and bart - Wiki.js API integration for incident documentation - Prometheus probe for Spirit gateway health - Safety guardrails: protected services, rate limiting, escalation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-19 10:36:20 -05:00
parent 59f529e3a7
commit 9943fd4410
5 changed files with 1045 additions and 2 deletions
@@ -0,0 +1,724 @@
+# Spirit Infrastructure Ops Agent Implementation Plan
+
+> **For agentic workers:** REQUIRED: Use superpowers:subagent-driven-development (if subagents available) or superpowers:executing-plans to implement this plan. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Repurpose the idle Spirit OpenClaw agent as an autonomous infrastructure ops agent that monitors Matrix alerts and takes corrective action.
+
+**Architecture:** Spirit runs in a custom Docker image with ops tooling (docker-cli, SSH), monitors the #notifications Matrix room for alerts from @alertbot, investigates issues via Docker socket (local) and SSH (remote nodes), takes corrective action, reports to the room, and documents incidents in Wiki.js.
+
+**Tech Stack:** Docker, OpenClaw, Matrix, SSH, Wiki.js GraphQL API
+
+**Spec:** `docs/superpowers/specs/2026-03-18-spirit-infra-ops-design.md`
+
+---
+
+## Chunk 1: Image, SSH, and Compose
+
+### Task 1: Build spirit-ops Docker image on hf-pdocker-01
+
+**Context:** Spirit currently uses `margaret-openclaw:latest` (node:22-bookworm-slim + openclaw). We need a new image that adds Docker CLI, SSH client, and diagnostic tools. Build on hf-pdocker-01 since that's where agents run.
+
+**Files:**
+- Create: `/tmp/spirit-ops-build/Dockerfile` (on hf-pdocker-01 via docker-swarm-ops agent)
+
+- [ ] **Step 1: Create the Dockerfile on hf-pdocker-01**
+
+Use the docker-swarm-ops agent to SSH to hf-pdocker-01 and create the build directory and Dockerfile:
+
+```dockerfile
+FROM margaret-openclaw:latest
+
+# Copy Docker CLI from official image (no daemon needed — uses mounted socket)
+COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
+
+# Install ops tooling
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    openssh-client \
+    curl \
+    jq \
+    procps \
+    net-tools \
+    iputils-ping \
+    dnsutils \
+    && rm -rf /var/lib/apt/lists/*
+```
+
+Commands:
+```bash
+ssh glyptic@192.168.86.192 "mkdir -p /tmp/spirit-ops-build"
+# SCP or write the Dockerfile
+ssh glyptic@192.168.86.192 "cat > /tmp/spirit-ops-build/Dockerfile << 'DOCKERFILE'
+FROM margaret-openclaw:latest
+
+COPY --from=docker:cli /usr/local/bin/docker /usr/local/bin/docker
+
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    openssh-client \
+    curl \
+    jq \
+    procps \
+    net-tools \
+    iputils-ping \
+    dnsutils \
+    && rm -rf /var/lib/apt/lists/*
+DOCKERFILE"
+```
+
+- [ ] **Step 2: Build the image**
+
+```bash
+ssh glyptic@192.168.86.192 "cd /tmp/spirit-ops-build && docker build --no-cache -t spirit-ops:latest ."
+```
+
+Expected: Image builds successfully. Docker CLI is copied as a static binary from the official `docker:cli` image. The existing OpenClaw + Matrix plugin from the base image is preserved.
+
+- [ ] **Step 3: Verify the image has the required tools**
+
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm spirit-ops:latest sh -c 'docker --version && ssh -V && curl --version | head -1 && jq --version && ps --version && ping -c1 127.0.0.1 > /dev/null && echo ALL OK'"
+```
+
+Expected: All tool versions print, ending with `ALL OK`.
+
+---
+
+### Task 2: Generate SSH keypair and deploy to nodes
+
+**Context:** Spirit needs passwordless SSH access to all 3 nodes as user `glyptic`. Keys are stored on NAS at `/mnt/infrastructure/spirit/ssh/` and mounted into the container at `/root/.ssh`. We pre-populate `known_hosts` with real host keys (no StrictHostKeyChecking=no).
+
+- [ ] **Step 1: Create the ssh directory on NAS**
+
+```bash
+ssh glyptic@192.168.86.192 "sudo mkdir -p /mnt/infrastructure/spirit/ssh && sudo chown root:root /mnt/infrastructure/spirit/ssh && sudo chmod 700 /mnt/infrastructure/spirit/ssh"
+```
+
+Note: NFS root_squash may prevent chown. Use the busybox workaround if needed:
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'mkdir -p /data/ssh && chmod 700 /data/ssh'"
+```
+
+- [ ] **Step 2: Generate ED25519 keypair**
+
+Generate the key on hf-pdocker-01 and copy to the NAS SSH directory:
+
+```bash
+ssh glyptic@192.168.86.192 "ssh-keygen -t ed25519 -f /tmp/spirit-ssh-key -N '' -C spirit-ops@homelab && sudo cp /tmp/spirit-ssh-key /mnt/infrastructure/spirit/ssh/id_ed25519 && sudo cp /tmp/spirit-ssh-key.pub /mnt/infrastructure/spirit/ssh/id_ed25519.pub && sudo chmod 600 /mnt/infrastructure/spirit/ssh/id_ed25519 && sudo chmod 644 /mnt/infrastructure/spirit/ssh/id_ed25519.pub && rm /tmp/spirit-ssh-key /tmp/spirit-ssh-key.pub"
+```
+
+Expected: `/mnt/infrastructure/spirit/ssh/id_ed25519` and `.pub` files exist with correct permissions.
+
+- [ ] **Step 3: Create SSH config file**
+
+```bash
+ssh glyptic@192.168.86.192 "sudo tee /mnt/infrastructure/spirit/ssh/config > /dev/null << 'EOF'
+Host hf-pdocker-01
+    HostName 192.168.86.192
+    User glyptic
+    IdentityFile /root/.ssh/id_ed25519
+
+Host hf-pdocker-02
+    HostName 192.168.86.100
+    User glyptic
+    IdentityFile /root/.ssh/id_ed25519
+
+Host bart
+    HostName 192.168.86.167
+    User glyptic
+    IdentityFile /root/.ssh/id_ed25519
+
+Host *
+    ServerAliveInterval 30
+    ServerAliveCountMax 3
+    UserKnownHostsFile /root/.ssh/known_hosts
+EOF"
+ssh glyptic@192.168.86.192 "sudo chmod 600 /mnt/infrastructure/spirit/ssh/config"
+```
+
+- [ ] **Step 4: Collect host keys and create known_hosts**
+
+```bash
+ssh glyptic@192.168.86.192 "ssh-keyscan -H 192.168.86.192 192.168.86.100 192.168.86.167 2>/dev/null | sudo tee /mnt/infrastructure/spirit/ssh/known_hosts > /dev/null && sudo chmod 644 /mnt/infrastructure/spirit/ssh/known_hosts"
+```
+
+Expected: `known_hosts` file contains SSH host keys for all 3 nodes.
+
+- [ ] **Step 5: Deploy public key to hf-pdocker-02 and bart**
+
+Read the public key first:
+```bash
+ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub"
+```
+
+Then append to authorized_keys on each remote node:
+```bash
+# hf-pdocker-02
+ssh glyptic@192.168.86.100 "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
+ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub" | ssh glyptic@192.168.86.100 "cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
+
+# bart
+ssh glyptic@192.168.86.167 "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
+ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub" | ssh glyptic@192.168.86.167 "cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
+```
+
+Also deploy to hf-pdocker-01 itself (Spirit may SSH to localhost):
+```bash
+ssh glyptic@192.168.86.192 "cat /mnt/infrastructure/spirit/ssh/id_ed25519.pub >> ~/.ssh/authorized_keys"
+```
+
+- [ ] **Step 6: Verify SSH connectivity from a test container**
+
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm -v /mnt/infrastructure/spirit/ssh:/root/.ssh:ro spirit-ops:latest sh -c 'ssh -o BatchMode=yes glyptic@192.168.86.100 hostname && ssh -o BatchMode=yes glyptic@192.168.86.167 hostname && echo SSH OK'"
+```
+
+Expected: Prints `hf-pdocker-02`, `bart`, then `SSH OK`.
+
+---
+
+### Task 3: Update compose file
+
+**Context:** Modify the `spirit` service in the ai-agents compose file to use the new image, mount docker socket and SSH keys, increase memory, and add the Wiki.js API key env var.
+
+**Files:**
+- Modify: `compose/hf-pdocker-01/ai-agents/docker-compose.yml:26-48`
+
+- [ ] **Step 1: Update the spirit service in docker-compose.yml**
+
+Change the spirit service block (lines 26-48) to:
+
+```yaml
+  spirit:
+    image: spirit-ops:latest
+    container_name: spirit
+    restart: unless-stopped
+    network_mode: host
+    user: "0:0"
+    labels:
+      - com.centurylinklabs.watchtower.enable=false
+    environment:
+      - NODE_ENV=production
+      - HOME=/home/glyptic
+      - USER=glyptic
+      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
+      - WIKIJS_API_KEY=${WIKIJS_API_KEY}
+      - OPENCLAW_STATE_DIR=/home/glyptic/.spirit
+      - OPENCLAW_CONFIG_PATH=/home/glyptic/.spirit/openclaw.json
+    volumes:
+      - /mnt/infrastructure/spirit:/home/glyptic/.spirit
+      - /var/run/docker.sock:/var/run/docker.sock
+      - /mnt/infrastructure/spirit/ssh:/root/.ssh:ro
+    command: ["sh", "-c", "mkdir -p /home/glyptic && node --max-old-space-size=1500 /usr/local/lib/node_modules/openclaw/dist/index.js gateway --allow-unconfigured"]
+    mem_limit: 3G
+    memswap_limit: 3G
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "50m"
+        max-file: "3"
+```
+
+- [ ] **Step 2: Add WIKIJS_API_KEY to the .env file on NAS**
+
+The .env file is at `/mnt/docker-data/compose/hf-pdocker-01/ai-agents/.env` on the NAS. Retrieve the Wiki.js API key from the documentation-keeper agent config or Wiki.js admin panel, then append:
+
+```bash
+ssh glyptic@192.168.86.192 "echo 'WIKIJS_API_KEY=<WIKIJS_JWT_TOKEN>' | sudo tee -a /mnt/docker-data/compose/hf-pdocker-01/ai-agents/.env > /dev/null"
+```
+
+The JWT token is the long-lived API key from Wiki.js (Administration > API Access). It's already used by the documentation-keeper agent.
+
+---
+
+## Chunk 2: Workspace Identity Files and Model Config
+
+### Task 4: Update Spirit's openclaw.json model to Claude
+
+**Context:** Spirit currently uses `ollama/qwen3.5` as primary model. For an infrastructure ops agent that needs strong reasoning and tool use, Claude is required. Update the primary model to `anthropic/claude-sonnet-4-5-20250514` (good balance of speed and capability for ops tasks — opus is overkill for container restarts).
+
+- [ ] **Step 1: Update the model in openclaw.json**
+
+Use the NFS write workaround since openclaw.json is root-owned:
+
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data alpine sh -c \"apk add --no-cache jq > /dev/null 2>&1 && jq '.models.primary = \"anthropic/claude-sonnet-4-5-20250514\"' /data/openclaw.json > /data/openclaw.json.tmp && mv /data/openclaw.json.tmp /data/openclaw.json\""
+```
+
+- [ ] **Step 2: Verify the change**
+
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox cat /data/openclaw.json" | grep -o '"primary":"[^"]*"'
+```
+
+Expected: `"primary":"anthropic/claude-sonnet-4-5-20250514"`
+
+---
+
+### Task 5: Write SOUL.md workspace identity file
+
+**Context:** Spirit's current SOUL.md is stale (references old hostname "Peggy"). Replace with infrastructure ops identity. The file is glyptic-owned so may be writable directly, but use the NFS workaround if needed.
+
+- [ ] **Step 1: Write SOUL.md to NAS**
+
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'cat > /data/workspace/SOUL.md << \"SOULEOF\"
+# Spirit — Infrastructure Operations Agent
+
+You are Spirit, an autonomous infrastructure operations agent for a homelab cluster. You monitor the #notifications Matrix room for alerts from @alertbot and take corrective action to keep services running.
+
+## Core Principles
+
+1. **Service availability is your top priority.** When an alert fires, investigate immediately and take action to restore service.
+2. **Always report what you find and what you do.** Post your investigation results and any actions taken back to the #notifications Matrix room so the owner can see what happened.
+3. **Document everything.** After taking corrective action, update the relevant Wiki.js page with an incident entry.
+4. **Respect the guardrails.** Some services are protected and must not be restarted. Rate-limit your actions. Escalate when remediation fails.
+5. **Data safety matters.** Never delete volumes, drop databases, or run destructive commands. Prefer restarts and cleanup over risky operations.
+
+## How You Work
+
+- You receive alerts as Matrix messages from @alertbot:hobbs.farm in the #notifications:hobbs.farm room
+- You have Docker CLI access on hf-pdocker-01 (local socket) and SSH access to hf-pdocker-02 and bart
+- You can run any diagnostic or remediation command on any node
+- You post your findings and actions back to the same Matrix room
+- You update Wiki.js documentation after taking corrective action
+SOULEOF'"
+```
+
+---
+
+### Task 6: Write IDENTITY.md workspace identity file
+
+**Context:** This is the comprehensive operational knowledge file. It tells Spirit everything about the cluster, alerts, remediation procedures, and guardrails.
+
+- [ ] **Step 1: Write IDENTITY.md to NAS**
+
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'cat > /data/workspace/IDENTITY.md << \"IDEOF\"
+# Infrastructure Operations Reference
+
+## Cluster Architecture
+
+### Nodes
+
+| Node | IP | RAM | Role | Access Method |
+|------|-----|-----|------|---------------|
+| hf-pdocker-01 | 192.168.86.192 | 23GB | Swarm Manager, primary services | Local Docker socket |
+| hf-pdocker-02 | 192.168.86.100 | 7.6GB | Dev tools, utilities | SSH as glyptic |
+| bart | 192.168.86.167 | 15GB | Media stack, Onyx | SSH as glyptic |
+| Synology NAS | 192.168.86.30 | - | NFS storage for all services | Not directly accessible |
+
+### Services by Node
+
+**hf-pdocker-01 (192.168.86.192):**
+- Frigate NVR (:5000) — 13 cameras, OpenVINO GPU detector
+- Frigate-Notify v2 (:5199) — MQTT→AI→Matrix notifications
+- clip-service (:5200) — CLIP embeddings
+- Open WebUI (:3010) — AI chat UI
+- Home Assistant (:8123) — home automation
+- ESPHome (:6052) — ESP device management
+- Mosquitto (:1883, :8883) — MQTT broker
+- Matrix/Synapse (:8008) — chat server
+- Well-known (:8010) — Matrix federation
+- Guacamole (:8199) — remote desktop gateway
+- AI Agents: margaret(:18790), spirit(:18795/you), clippy(:18800), bob(:18810), travis(:18815)
+- Traefik (:80, :443, :8090) — reverse proxy
+- AdGuard Home (:3000, :53) — DNS
+- Grafana (:3001), Prometheus (:9095), Alertmanager (:9093)
+- InfluxDB (:8086), Blackbox Exporter (:9115)
+- matrix-alertmanager-receiver (:9094) — alert→Matrix bridge
+- Farm Manager (:8888), Farm Agent (:8889)
+
+**hf-pdocker-02 (192.168.86.100):**
+- Unifi (:8443) — network controller
+- Gitea (:3003) — git server
+- Semaphore (:3004) — Ansible UI
+- Wiki.js (:3002) — documentation
+- Vaultwarden (:26226) — password manager
+- Actual Budget (:5006) — finance
+- MCPO (:8001), Trivy (:4954)
+- SearXNG (:8888) — meta-search
+- Unstructured API (:8000) — document parsing
+- n8n + n8n-db — workflow automation
+- NetClaw/Aaron (:1179) — AI network agent
+- Farm Agent (:8889)
+- Watchtower — auto-updates
+
+**bart (192.168.86.167):**
+- Jellyfin (:8096) — media server
+- Sonarr (:8989), Radarr (:7878) — media management
+- SABnzbd (:8082), NZBHydra2 (:5076) — downloads
+- Jellyseerr (:5055) — media requests
+- OpenVAS (:9392) — vulnerability scanner
+- Onyx (:3005) — AI document search/RAG
+- Farm Agent (:8889)
+- Watchtower — auto-updates
+
+### Storage
+
+All service data on NAS via NFS at /mnt/docker-data/ on every node:
+- configs/{service}/ — application configs
+- databases/{service}-db/ — database volumes
+- compose/{node}/{service}/ — compose files
+
+### Compose File Locations
+
+To manage services, the compose files are at:
+- hf-pdocker-01: /mnt/docker-data/compose/hf-pdocker-01/{service}/docker-compose.yml
+- hf-pdocker-02: /mnt/docker-data/compose/hf-pdocker-02/{service}/docker-compose.yml
+- bart: /mnt/docker-data/compose/bart/{service}/docker-compose.yml
+
+To restart a compose service on a remote node:
+```
+ssh glyptic@{node-ip} "cd /mnt/docker-data/compose/{node}/{service} && docker compose up -d"
+```
+
+## Commands
+
+### Local (hf-pdocker-01 — Docker socket)
+```bash
+docker ps                          # list running containers
+docker ps -a                       # list all containers including stopped
+docker logs <container> --tail 100 # view container logs
+docker restart <container>         # restart a container
+docker stats --no-stream           # resource usage snapshot
+docker system df                   # disk usage by Docker
+docker system prune -f             # clean up unused images/containers (NOT volumes)
+docker compose -f /path/to/docker-compose.yml up -d  # redeploy a service
+```
+
+### Remote (hf-pdocker-02, bart — via SSH)
+```bash
+ssh glyptic@<node-ip> "docker ps"
+ssh glyptic@<node-ip> "docker logs <container> --tail 100"
+ssh glyptic@<node-ip> "docker restart <container>"
+ssh glyptic@<node-ip> "df -h"
+ssh glyptic@<node-ip> "free -m"
+ssh glyptic@<node-ip> "cd /mnt/docker-data/compose/<node>/<service> && docker compose up -d"
+```
+
+### System Diagnostics
+```bash
+df -h                              # disk usage
+free -m                            # memory usage
+top -bn1 | head -20                # CPU/process snapshot
+uptime                             # load averages
+mount | grep nfs                   # NFS mount status
+ping -c3 192.168.86.30             # NAS reachability
+```
+
+## Safety Guardrails
+
+### Protected Services — NEVER restart these without explicit owner request
+- **Monitoring pipeline**: prometheus, alertmanager, matrix-alertmanager-receiver, grafana, blackbox-exporter
+- **Network infrastructure**: traefik, adguard
+- **AI agents**: margaret, spirit (yourself), bob, clippy, travis
+- **All database containers**: synapse-db, guacamole-db, wikijs-db, gitea-db, semaphore-db, unifi-db, n8n-db, any container ending in -db
+
+If a protected service is down, INVESTIGATE and REPORT but do NOT restart. Tell the owner what you found.
+
+### Rate Limiting
+- Do NOT restart the same container more than once per 30 minutes
+- Maximum 5 remediation actions per hour across all containers
+- After 2 failed remediation attempts for the same alert, STOP and escalate to the owner with full diagnostics
+
+### Rollback Awareness
+- After restarting a container, wait 5 minutes. If the same alert fires again, do NOT restart again — report the persistent failure with full logs
+- NEVER run: docker system prune --volumes, docker volume rm, rm -rf on data directories
+- docker system prune -f (without --volumes) IS allowed for disk cleanup
+
+## Alert Response Procedures
+
+### Response Flow
+1. Parse the alert message from @alertbot — extract: alert name, severity, instance, description
+2. Investigate — run relevant diagnostic commands
+3. Remediate — take corrective action (if the service is not protected and rate limits allow)
+4. Report — post findings and actions to #notifications room
+5. Document — update the relevant Wiki.js page
+
+### Alert-Specific Procedures
+
+| Alert | Severity | What to Check | What to Do |
+|-------|----------|---------------|------------|
+| ContainerDown | warning | docker ps -a, docker logs | docker restart or docker compose up -d |
+| HighDiskUsage | critical | df -h, docker system df | docker system prune -f, identify large files |
+| DiskSpaceWarning | warning | df -h, docker system df | Report findings, suggest cleanup |
+| HighMemoryUsage | warning | free -m, docker stats --no-stream | Identify top memory consumer, restart if non-protected |
+| CriticalMemoryUsage | critical | free -m, docker stats --no-stream | Restart heaviest non-protected container |
+| HighCPUUsage | warning | top -bn1, docker stats --no-stream | Identify CPU hog, report |
+| CriticalCPUUsage | critical | top -bn1, docker stats --no-stream | Restart CPU hog if non-protected |
+| HighSystemLoad | warning | uptime, top -bn1, docker stats | Report load source |
+| NFSMountDown | critical | mount, ping 192.168.86.30 | Try sudo mount -a, report if NAS unreachable |
+| NASDiskSpaceLow | warning | df -h /mnt/docker-data | Report, identify large consumers |
+| SlowHTTPResponse | warning | curl timing, docker logs | Restart slow service if non-protected |
+| FrigateHighMemory | warning | docker stats frigate | docker restart frigate |
+| ServiceHighMemory | warning | docker stats | Restart container if non-protected |
+| ServerDown | critical | ping, SSH attempt | Report — cannot fix hardware |
+| ServerRebooted | info | uptime, docker ps | Report, verify all containers running |
+| HighSwapUsage | warning | free -m, docker stats | Identify swap contributor |
+| CriticalSwapUsage | critical | free -m, docker stats | Restart heaviest swap user if non-protected |
+| PrometheusTargetDown | warning | curl the target, docker ps | Restart target if down and non-protected |
+| Watchdog | info | N/A | Ignore — dead-man switch |
+| UnifiDeviceDown | warning | Check Unifi controller UI | Report — network device issue |
+
+### Resolved Alerts
+When you see a RESOLVED alert from @alertbot, acknowledge it briefly. No investigation needed.
+
+### Unknown Alerts
+For any alert type not listed above: investigate and report findings, but do NOT take remediation action.
+
+## Wiki.js Documentation
+
+After taking any corrective action, update the relevant Wiki.js page.
+
+**API endpoint**: http://192.168.86.100:3002/graphql
+**Auth**: Bearer token from WIKIJS_API_KEY environment variable
+
+**Key pages:**
+- infrastructure/overview (ID 118)
+- infrastructure/service-access (ID 119)
+- infrastructure/maintenance (ID 120)
+- infrastructure/alerting (ID 121)
+
+**To read a page:**
+```bash
+curl -s http://192.168.86.100:3002/graphql \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer $WIKIJS_API_KEY" \
+  -d '"'"'{"query":"{ pages { single(id: 120) { id title content } } }"}'"'"'
+```
+
+**To update a page** (append incident entry to existing content):
+1. Read current page content
+2. Append your incident entry at the bottom
+3. Update the page with the full new content
+
+Incident entry format:
+```markdown
+### YYYY-MM-DD HH:MM — [Alert Name] ([severity])
+- **Node**: [affected node]
+- **Service**: [affected service]
+- **Root cause**: [what you found]
+- **Action taken**: [what you did]
+- **Outcome**: [result — resolved/escalated/persistent]
+```
+IDEOF'"
+```
+
+Note: This is a large heredoc. If it fails due to shell escaping issues, use the base64 approach instead:
+1. Write IDENTITY.md content to a local temp file
+2. Base64 encode and pipe to the target:
+```bash
+base64 /tmp/IDENTITY.md | ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data -i busybox sh -c 'base64 -d > /data/workspace/IDENTITY.md'"
+```
+
+Or write locally and SCP directly:
+```bash
+scp /tmp/IDENTITY.md glyptic@192.168.86.192:/tmp/IDENTITY.md
+ssh glyptic@192.168.86.192 "sudo cp /tmp/IDENTITY.md /mnt/infrastructure/spirit/workspace/IDENTITY.md"
+```
+
+---
+
+### Task 7: Write USER.md workspace identity file
+
+- [ ] **Step 1: Write USER.md to NAS**
+
+```bash
+ssh glyptic@192.168.86.192 "docker run --rm -u 0:0 -v /mnt/infrastructure/spirit:/data busybox sh -c 'cat > /data/workspace/USER.md << \"USEREOF\"
+# Owner: Jeremy
+
+- Homelab operator, full infrastructure access
+- Prefers autonomous action — fix things first, report after
+- Full trust for corrective actions within the safety guardrails
+- Timezone: US Central (CST/CDT)
+- Reachable via Matrix: @jeremy:hobbs.farm
+- If something needs human attention, post in #notifications — Jeremy monitors it
+USEREOF'"
+```
+
+---
+
+## Chunk 3: Deploy and Verify
+
+### Task 8: Deploy Spirit with new configuration
+
+**Context:** SCP the updated compose file to hf-pdocker-01, then redeploy only the Spirit container. Other agents (margaret, clippy, bob, travis) remain untouched.
+
+- [ ] **Step 1: SCP the updated compose file to the NAS**
+
+```bash
+scp compose/hf-pdocker-01/ai-agents/docker-compose.yml glyptic@192.168.86.192:/mnt/docker-data/compose/hf-pdocker-01/ai-agents/docker-compose.yml
+```
+
+- [ ] **Step 2: Stop only the Spirit container**
+
+```bash
+ssh glyptic@192.168.86.192 "docker stop spirit && docker rm spirit"
+```
+
+- [ ] **Step 3: Redeploy Spirit using docker compose**
+
+```bash
+ssh glyptic@192.168.86.192 "cd /mnt/docker-data/compose/hf-pdocker-01/ai-agents && docker compose up -d spirit"
+```
+
+Expected: Only the `spirit` container is recreated. Other agents remain running.
+
+- [ ] **Step 4: Verify Spirit container is running with correct image and mounts**
+
+```bash
+ssh glyptic@192.168.86.192 "docker inspect spirit --format '{{.Config.Image}} {{.HostConfig.Memory}} {{range .Mounts}}{{.Source}}->{{.Destination}} {{end}}'"
+```
+
+Expected: Image is `spirit-ops:latest`, memory is 3221225472 (3GB), mounts include `/var/run/docker.sock` and `/mnt/infrastructure/spirit/ssh`.
+
+- [ ] **Step 5: Verify Docker socket works inside Spirit**
+
+```bash
+ssh glyptic@192.168.86.192 "docker exec spirit docker ps --format 'table {{.Names}}\t{{.Status}}' | head -5"
+```
+
+Expected: Lists running containers on hf-pdocker-01.
+
+- [ ] **Step 6: Verify SSH works inside Spirit**
+
+```bash
+ssh glyptic@192.168.86.192 "docker exec spirit ssh -o BatchMode=yes glyptic@192.168.86.100 hostname"
+ssh glyptic@192.168.86.192 "docker exec spirit ssh -o BatchMode=yes glyptic@192.168.86.167 hostname"
+```
+
+Expected: Prints `hf-pdocker-02` and `bart`.
+
+- [ ] **Step 7: Wait for Spirit's Matrix plugin to initialize (90-120 seconds)**
+
+```bash
+ssh glyptic@192.168.86.192 "docker logs spirit --tail 20"
+```
+
+Expected: After 90-120 seconds, logs show Matrix plugin connected. No output during crypto init is normal.
+
+---
+
+### Task 9: Invite Spirit to #notifications room
+
+**Context:** Use the Synapse admin API to invite @spirit:hobbs.farm to the notifications room. Spirit must already be running so the Matrix plugin accepts the invite.
+
+- [ ] **Step 1: Invite Spirit to the room using Synapse admin API**
+
+Use Spirit's own admin token (Spirit is a Synapse admin):
+
+```bash
+ssh glyptic@192.168.86.192 "curl -s -X POST 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/invite' -H 'Authorization: Bearer syt_c3Bpcml0_AUBChFVZIEcXzOjUnNHo_0aaMzg' -H 'Content-Type: application/json' -d '{\"user_id\": \"@spirit:hobbs.farm\"}'"
+```
+
+Note: If Spirit can't invite itself, use the alertbot token or another admin account:
+```bash
+ssh glyptic@192.168.86.192 "curl -s -X POST 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/invite' -H 'Authorization: Bearer syt_YWxlcnRib3Q_eOVAYjYKWBZxymfdxhPE_0iYlOZ' -H 'Content-Type: application/json' -d '{\"user_id\": \"@spirit:hobbs.farm\"}'"
+```
+
+Expected: `{}` or `{"room_id":"!jYmqxbzEsvRgKYvwMP:hobbs.farm"}` — success.
+
+- [ ] **Step 2: Verify Spirit joined the room**
+
+Wait 10-30 seconds for the OpenClaw Matrix plugin to auto-accept, then check:
+
+```bash
+ssh glyptic@192.168.86.192 "curl -s 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/joined_members' -H 'Authorization: Bearer syt_c3Bpcml0_AUBChFVZIEcXzOjUnNHo_0aaMzg' | jq '.joined | keys'"
+```
+
+Expected: List includes `@spirit:hobbs.farm` alongside `@alertbot:hobbs.farm`.
+
+---
+
+### Task 10: End-to-end test with a real alert
+
+**Context:** Send a test alert through Alertmanager to trigger the full pipeline: Alertmanager → matrix-alertmanager-receiver → @alertbot → #notifications → Spirit sees it and responds.
+
+- [ ] **Step 1: Send a test alert via Alertmanager API**
+
+```bash
+ssh glyptic@192.168.86.192 "curl -s -X POST http://192.168.86.192:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[{
+  \"labels\": {
+    \"alertname\": \"TestAlert\",
+    \"severity\": \"info\",
+    \"instance\": \"test-instance:9999\",
+    \"job\": \"test\"
+  },
+  \"annotations\": {
+    \"summary\": \"Test alert for Spirit ops agent verification\",
+    \"description\": \"This is a test alert. Spirit should acknowledge it and report in the notifications channel.\"
+  },
+  \"generatorURL\": \"http://192.168.86.192:9095\"
+}]'"
+```
+
+Expected: HTTP 200. Alert flows through the pipeline.
+
+- [ ] **Step 2: Verify Spirit responds in the Matrix room**
+
+Wait 1-2 minutes, then check recent messages in the room:
+
+```bash
+ssh glyptic@192.168.86.192 "curl -s 'http://192.168.86.192:8008/_matrix/client/v3/rooms/!jYmqxbzEsvRgKYvwMP:hobbs.farm/messages?dir=b&limit=5' -H 'Authorization: Bearer syt_c3Bpcml0_AUBChFVZIEcXzOjUnNHo_0aaMzg' | jq '.chunk[] | {sender: .sender, body: .content.body}'"
+```
+
+Expected: Messages from both `@alertbot:hobbs.farm` (the alert) and `@spirit:hobbs.farm` (the response/acknowledgement).
+
+- [ ] **Step 3: Resolve the test alert**
+
+```bash
+ssh glyptic@192.168.86.192 "curl -s -X POST http://192.168.86.192:9093/api/v2/alerts -H 'Content-Type: application/json' -d '[{
+  \"labels\": {
+    \"alertname\": \"TestAlert\",
+    \"severity\": \"info\",
+    \"instance\": \"test-instance:9999\",
+    \"job\": \"test\"
+  },
+  \"annotations\": {
+    \"summary\": \"Test alert for Spirit ops agent verification\",
+    \"description\": \"This is a test alert.\"
+  },
+  \"endsAt\": \"2026-03-19T00:00:00Z\",
+  \"generatorURL\": \"http://192.168.86.192:9095\"
+}]'"
+```
+
+Expected: Alertmanager sends a "resolved" notification. Spirit acknowledges it briefly.
+
+---
+
+### Task 11: Post-deployment updates
+
+- [ ] **Step 1: Commit compose changes and spec to git**
+
+```bash
+git add compose/hf-pdocker-01/ai-agents/docker-compose.yml
+git add docs/superpowers/specs/2026-03-18-spirit-infra-ops-design.md
+git add docs/superpowers/plans/2026-03-19-spirit-infra-ops.md
+git commit -m "infra: repurpose Spirit as infrastructure ops agent with alert monitoring"
+```
+
+- [ ] **Step 2: Update CLAUDE.md**
+
+Update the Spirit entry in CLAUDE.md's service distribution to reflect its new role. Change the AI Agents line or add a note that Spirit is now the infrastructure ops agent monitoring #notifications.
+
+- [ ] **Step 3: Update MEMORY.md**
+
+Add a memory entry for Spirit's new role as infrastructure ops agent.
+
+- [ ] **Step 4: Add Prometheus probe for Spirit**
+
+Spirit is now a critical infrastructure component. Add `http://192.168.86.192:18795` to the `http-pdocker01` job in `stacks/monitoring/prometheus.yml`, then SCP and hot-reload:
+
+```bash
+# After editing prometheus.yml locally:
+scp stacks/monitoring/prometheus.yml glyptic@192.168.86.192:/mnt/docker-data/configs/prometheus/prometheus.yml
+ssh glyptic@192.168.86.192 "curl -X POST http://192.168.86.192:9095/-/reload"
+```
+
+- [ ] **Step 5: Push to Gitea**
+
+Use the gitea-backup agent to commit and push all changes.
+
+- [ ] **Step 6: Update Wiki.js**
+
+Use the documentation-keeper agent to update the infrastructure/alerting wiki page (ID 121) documenting that Spirit now monitors and responds to alerts in #notifications.