hobbs-iac/docs/superpowers/specs/2026-03-18-spirit-infra-ops-design.md

Spirit Infrastructure Ops Agent

Overview

Repurpose the idle Spirit OpenClaw agent (@spirit:hobbs.farm) as an autonomous infrastructure operations agent. Spirit joins the #notifications:hobbs.farm Matrix room, monitors alerts posted by @alertbot:hobbs.farm, investigates issues, takes corrective action, and reports findings back to the room.

Architecture

Prometheus → Alertmanager → @alertbot → #notifications room
                                              ↓
                                    @spirit reads alert
                                              ↓
                                    Investigate (docker/SSH)
                                              ↓
                                    Remediate + report to room

Spirit has full autonomy to take corrective action, subject to the safety guardrails defined below.

Cluster Context

Spirit must have complete knowledge of the infrastructure:

• hf-pdocker-01 (192.168.86.192) — Swarm Manager, 23GB RAM. Runs: Frigate, Frigate-Notify, clip-service, Open WebUI, Home Assistant, ESPHome, Mosquitto, Matrix/Synapse, Guacamole, AI Agents, Traefik, AdGuard, Monitoring stack, Farm Manager (:8888), Farm Agent (:8889). Docker socket available locally.
• hf-pdocker-02 (192.168.86.100) — 7.6GB RAM. Runs: Unifi, Gitea, Semaphore, Wiki.js, Vaultwarden, Actual Budget, MCPO, Trivy, SearXNG, Unstructured API, n8n, NetClaw, Farm Agent, Watchtower. Access via SSH.
• bart (192.168.86.167) — 15GB RAM. Runs: Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr, OpenVAS, Onyx, Farm Agent, Watchtower. Access via SSH.
• Synology NAS (192.168.86.30) — NFS storage at /mnt/docker-data on all nodes.

Components

1. Custom Docker Image: spirit-ops:latest

Build location: /tmp/spirit-ops-build/ on hf-pdocker-01.

Extends margaret-openclaw:latest with operational tooling:

• docker-cli (Docker CLI, no daemon — uses mounted socket)
• openssh-client (SSH to remote nodes)
• curl, jq (API calls, JSON parsing)
• procps, net-tools, iputils-ping, dnsutils (system diagnostics)

The Dockerfile installs these packages on top of the existing OpenClaw + Matrix plugin base.

2. Compose Changes

In compose/hf-pdocker-01/ai-agents/docker-compose.yml, modify the spirit service:

• Change image from margaret-openclaw:latest to spirit-ops:latest
• Add volume mount: /var/run/docker.sock:/var/run/docker.sock (local Docker access)
• Add volume mount: /mnt/infrastructure/spirit/ssh:/root/.ssh:ro (SSH keys — mounted to /root/.ssh since container runs as root)
• Increase mem_limit and memswap_limit to 3G (diagnostic commands + alert handling need headroom beyond the 1.5G Node.js allocation)
• Add environment variable WIKIJS_API_KEY (the long-lived JWT token for Wiki.js GraphQL API)
• All other settings (ports, network mode) remain unchanged.

Other agent services (margaret, clippy, bob, travis) are not modified.

3. SSH Key Setup

• Generate ED25519 keypair on hf-pdocker-01
• Store private key at /mnt/infrastructure/spirit/ssh/id_ed25519 (NAS, mounted into container)
• Store public key at /mnt/infrastructure/spirit/ssh/id_ed25519.pub
• Create /mnt/infrastructure/spirit/ssh/config with:
  • IdentityFile /root/.ssh/id_ed25519 (explicit path)
  • Pre-populated known_hosts file with actual host keys from all 3 nodes (no StrictHostKeyChecking=no)
• Deploy public key to ~glyptic/.ssh/authorized_keys on hf-pdocker-02 and bart
• SSH user: glyptic (has sudo on all nodes)

4. Workspace Identity Files

Rewrite Spirit's workspace files on NAS at /mnt/infrastructure/spirit/workspace/:

SOUL.md — Core identity:

• Spirit is an infrastructure operations agent for a homelab cluster
• Full autonomy to investigate and remediate alerts, subject to safety guardrails
• Always reports actions and findings to the Matrix room
• Prioritizes service availability and data safety

IDENTITY.md — Detailed operational knowledge:

• Complete cluster architecture (nodes, IPs, services, ports)
• Compose file locations on NAS (/mnt/docker-data/compose/{node}/{service}/)
• All alert types and remediation procedures (see table below)
• Docker commands for local ops, SSH commands for remote ops
• NFS mount details and known issues (root_squash, database user requirements)
• Safety guardrails and protected services list
• Escalation behavior and rate limiting rules

USER.md — Owner context:

• Jeremy, homelab operator
• Prefers autonomous action with post-action reporting
• Full trust for corrective actions within guardrails

5. Matrix Room Membership

• Invite @spirit:hobbs.farm to !jYmqxbzEsvRgKYvwMP:hobbs.farm using Synapse admin API
• Spirit's OpenClaw Matrix plugin auto-accepts room invites
• Spirit begins monitoring all messages in the room
• Spirit responds to alert messages from @alertbot:hobbs.farm
• Spirit distinguishes alerts (HTML-formatted messages from @alertbot with severity/color coding) from normal conversation

Safety Guardrails

Protected Services (never restart without explicit owner request)

• Prometheus, Alertmanager, matrix-alertmanager-receiver (restarting breaks the alerting pipeline)
• Traefik (restarting drops all external access)
• Spirit's own container and other AI agents
• Database containers (*-db, guacamole-db, etc.)

Rate Limiting

• Per-container cooldown: Do not restart the same container more than once per 30 minutes
• Global budget: Maximum 5 remediation actions per hour
• Escalation: After 2 failed remediation attempts for the same alert, stop trying and report to the owner with full diagnostics

Rollback Awareness

• After restarting a container, monitor for 5 minutes. If the same alert re-fires, do not attempt further restarts — report the persistent failure with logs.
• Never delete data, drop volumes, or run docker system prune --volumes
• docker system prune -f (without --volumes) is allowed for disk cleanup

Command Logging

• Report every command executed and its output to the Matrix room, either inline or as a summary after investigation

Wiki Documentation

• After taking any corrective action, Spirit updates the relevant Wiki.js page to document what happened, what was done, and when
• Wiki.js GraphQL API: http://192.168.86.100:3002/graphql (internal, from container via SSH or direct HTTP)
• Authentication: Bearer token (long-lived JWT API key, stored as env var WIKIJS_API_KEY in Spirit's compose)
• Spirit appends incident entries to the relevant wiki page (e.g., infrastructure/alerting page ID 121, or service-specific pages)
• Documentation format: timestamp, alert name, severity, root cause (if identified), actions taken, outcome
• Key wiki pages: infrastructure/overview (118), service-access (119), maintenance (120), alerting (121)
• Uses GraphQL mutations: pages { update(id, content, ...) } to append to existing pages

Alert Response Behavior

When Spirit sees an alert from @alertbot:

1. Parse the alert: extract alert name, severity, instance, description
2. Investigate: run relevant diagnostics (docker ps, logs, df, free, curl health endpoints)
3. Remediate: take corrective action based on alert type (respecting guardrails)
4. Report: post findings and actions taken back to #notifications
5. Document: update the relevant Wiki.js page with an incident entry (timestamp, alert, actions taken, outcome)

Remediation Procedures (All Alert Types)

Alert	Severity	Investigation	Remediation
ContainerDown	warning	docker ps -a, docker logs <container>	docker restart <container> or docker compose up -d
HighDiskUsage	critical	df -h, docker system df	docker system prune -f, clear old logs
DiskSpaceWarning	warning	df -h, docker system df	Report findings, suggest cleanup
HighMemoryUsage	warning	free -m, docker stats --no-stream	Identify memory hog, restart if needed
CriticalMemoryUsage	critical	Same as above	Restart heaviest non-protected container, report
HighCPUUsage	warning	top -bn1, docker stats --no-stream	Identify CPU hog, report
CriticalCPUUsage	critical	Same as above	Identify and restart CPU hog if safe
HighSystemLoad	warning	uptime, top -bn1, docker stats	Report findings, identify load source
NFSMountDown	critical	mount | grep nfs, ping 192.168.86.30	Attempt remount, report if NAS unreachable
NASDiskSpaceLow	warning	df -h /mnt/docker-data	Report, identify large consumers
SlowHTTPResponse	warning	curl -w timing, docker logs	Restart slow service
FrigateHighMemory	warning	docker stats frigate	docker restart frigate
ServiceHighMemory	warning	docker stats, identify container	Restart if safe
ServerDown	critical	SSH attempt, ping	Report — cannot fix hardware
ServerRebooted	info	Check uptime, verify services	Report, verify all containers are running
HighSwapUsage	warning	free -m, docker stats	Identify swap contributor, restart
CriticalSwapUsage	critical	Same as above	Restart heaviest swap user
PrometheusTargetDown	warning	curl the target, docker ps	Restart target service if down
Watchdog	info	N/A	Ignore (dead-man's switch, silenced in Alertmanager)
UnifiDeviceDown	warning	Check Unifi controller	Report — network device issue

Resolved Alerts

When Spirit sees a "resolved" alert, acknowledge briefly (no investigation needed).

Catch-All Rule

For any unrecognized alert type not in the table above: investigate and report findings, but do not take remediation action.

What Does Not Change

• Spirit's Matrix account, access token, homeserver config
• Spirit's gateway port (18795) and OpenClaw config structure
• Other agents (margaret, clippy, bob, travis)
• Alertbot continues posting alerts — Spirit is additive
• Prometheus, Alertmanager, alert rules — all unchanged

Implementation Steps

1. Build spirit-ops:latest image on hf-pdocker-01
2. Generate SSH keypair, collect host keys, deploy public key to nodes
3. Write workspace identity files (SOUL.md, IDENTITY.md, USER.md) to NAS
4. Update compose file (image, volume mounts, memory limit)
5. Redeploy Spirit container
6. Invite Spirit to #notifications room (after new container is running)
7. Verify: check SSH connectivity, docker socket access, Matrix room presence
8. Test: send a test alert through Alertmanager, confirm Spirit investigates and responds