infra: repurpose Spirit as autonomous infrastructure ops agent
Spirit now monitors #notifications Matrix room for Prometheus alerts, auto-investigates issues, takes corrective action (container restarts, disk cleanup, etc.), and documents incidents in Wiki.js. Changes: - Custom spirit-ops Docker image (Docker CLI + SSH + diagnostic tools) - Docker socket mount for local container management - SSH keys for remote access to hf-pdocker-02 and bart - Wiki.js API integration for incident documentation - Prometheus probe for Spirit gateway health - Safety guardrails: protected services, rate limiting, escalation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,191 @@
|
||||
# Spirit Infrastructure Ops Agent
|
||||
|
||||
## Overview
|
||||
|
||||
Repurpose the idle Spirit OpenClaw agent (`@spirit:hobbs.farm`) as an autonomous infrastructure operations agent. Spirit joins the `#notifications:hobbs.farm` Matrix room, monitors alerts posted by `@alertbot:hobbs.farm`, investigates issues, takes corrective action, and reports findings back to the room.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Prometheus → Alertmanager → @alertbot → #notifications room
|
||||
↓
|
||||
@spirit reads alert
|
||||
↓
|
||||
Investigate (docker/SSH)
|
||||
↓
|
||||
Remediate + report to room
|
||||
```
|
||||
|
||||
Spirit has full autonomy to take corrective action, subject to the safety guardrails defined below.
|
||||
|
||||
## Cluster Context
|
||||
|
||||
Spirit must have complete knowledge of the infrastructure:
|
||||
|
||||
- **hf-pdocker-01** (192.168.86.192) — Swarm Manager, 23GB RAM. Runs: Frigate, Frigate-Notify, clip-service, Open WebUI, Home Assistant, ESPHome, Mosquitto, Matrix/Synapse, Guacamole, AI Agents, Traefik, AdGuard, Monitoring stack, Farm Manager (:8888), Farm Agent (:8889). Docker socket available locally.
|
||||
- **hf-pdocker-02** (192.168.86.100) — 7.6GB RAM. Runs: Unifi, Gitea, Semaphore, Wiki.js, Vaultwarden, Actual Budget, MCPO, Trivy, SearXNG, Unstructured API, n8n, NetClaw, Farm Agent, Watchtower. Access via SSH.
|
||||
- **bart** (192.168.86.167) — 15GB RAM. Runs: Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr, OpenVAS, Onyx, Farm Agent, Watchtower. Access via SSH.
|
||||
- **Synology NAS** (192.168.86.30) — NFS storage at `/mnt/docker-data` on all nodes.
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Custom Docker Image: `spirit-ops:latest`
|
||||
|
||||
Build location: `/tmp/spirit-ops-build/` on hf-pdocker-01.
|
||||
|
||||
Extends `margaret-openclaw:latest` with operational tooling:
|
||||
|
||||
- `docker-cli` (Docker CLI, no daemon — uses mounted socket)
|
||||
- `openssh-client` (SSH to remote nodes)
|
||||
- `curl`, `jq` (API calls, JSON parsing)
|
||||
- `procps`, `net-tools`, `iputils-ping`, `dnsutils` (system diagnostics)
|
||||
|
||||
The Dockerfile installs these packages on top of the existing OpenClaw + Matrix plugin base.
|
||||
|
||||
### 2. Compose Changes
|
||||
|
||||
In `compose/hf-pdocker-01/ai-agents/docker-compose.yml`, modify the `spirit` service:
|
||||
|
||||
- Change image from `margaret-openclaw:latest` to `spirit-ops:latest`
|
||||
- Add volume mount: `/var/run/docker.sock:/var/run/docker.sock` (local Docker access)
|
||||
- Add volume mount: `/mnt/infrastructure/spirit/ssh:/root/.ssh:ro` (SSH keys — mounted to `/root/.ssh` since container runs as root)
|
||||
- Increase `mem_limit` and `memswap_limit` to `3G` (diagnostic commands + alert handling need headroom beyond the 1.5G Node.js allocation)
|
||||
- Add environment variable `WIKIJS_API_KEY` (the long-lived JWT token for Wiki.js GraphQL API)
|
||||
- All other settings (ports, network mode) remain unchanged.
|
||||
|
||||
Other agent services (margaret, clippy, bob, travis) are not modified.
|
||||
|
||||
### 3. SSH Key Setup
|
||||
|
||||
- Generate ED25519 keypair on hf-pdocker-01
|
||||
- Store private key at `/mnt/infrastructure/spirit/ssh/id_ed25519` (NAS, mounted into container)
|
||||
- Store public key at `/mnt/infrastructure/spirit/ssh/id_ed25519.pub`
|
||||
- Create `/mnt/infrastructure/spirit/ssh/config` with:
|
||||
- `IdentityFile /root/.ssh/id_ed25519` (explicit path)
|
||||
- Pre-populated `known_hosts` file with actual host keys from all 3 nodes (no `StrictHostKeyChecking=no`)
|
||||
- Deploy public key to `~glyptic/.ssh/authorized_keys` on hf-pdocker-02 and bart
|
||||
- SSH user: `glyptic` (has sudo on all nodes)
|
||||
|
||||
### 4. Workspace Identity Files
|
||||
|
||||
Rewrite Spirit's workspace files on NAS at `/mnt/infrastructure/spirit/workspace/`:
|
||||
|
||||
**SOUL.md** — Core identity:
|
||||
- Spirit is an infrastructure operations agent for a homelab cluster
|
||||
- Full autonomy to investigate and remediate alerts, subject to safety guardrails
|
||||
- Always reports actions and findings to the Matrix room
|
||||
- Prioritizes service availability and data safety
|
||||
|
||||
**IDENTITY.md** — Detailed operational knowledge:
|
||||
- Complete cluster architecture (nodes, IPs, services, ports)
|
||||
- Compose file locations on NAS (`/mnt/docker-data/compose/{node}/{service}/`)
|
||||
- All alert types and remediation procedures (see table below)
|
||||
- Docker commands for local ops, SSH commands for remote ops
|
||||
- NFS mount details and known issues (root_squash, database user requirements)
|
||||
- Safety guardrails and protected services list
|
||||
- Escalation behavior and rate limiting rules
|
||||
|
||||
**USER.md** — Owner context:
|
||||
- Jeremy, homelab operator
|
||||
- Prefers autonomous action with post-action reporting
|
||||
- Full trust for corrective actions within guardrails
|
||||
|
||||
### 5. Matrix Room Membership
|
||||
|
||||
- Invite `@spirit:hobbs.farm` to `!jYmqxbzEsvRgKYvwMP:hobbs.farm` using Synapse admin API
|
||||
- Spirit's OpenClaw Matrix plugin auto-accepts room invites
|
||||
- Spirit begins monitoring all messages in the room
|
||||
- Spirit responds to alert messages from `@alertbot:hobbs.farm`
|
||||
- Spirit distinguishes alerts (HTML-formatted messages from @alertbot with severity/color coding) from normal conversation
|
||||
|
||||
## Safety Guardrails
|
||||
|
||||
### Protected Services (never restart without explicit owner request)
|
||||
- Prometheus, Alertmanager, matrix-alertmanager-receiver (restarting breaks the alerting pipeline)
|
||||
- Traefik (restarting drops all external access)
|
||||
- Spirit's own container and other AI agents
|
||||
- Database containers (*-db, guacamole-db, etc.)
|
||||
|
||||
### Rate Limiting
|
||||
- **Per-container cooldown**: Do not restart the same container more than once per 30 minutes
|
||||
- **Global budget**: Maximum 5 remediation actions per hour
|
||||
- **Escalation**: After 2 failed remediation attempts for the same alert, stop trying and report to the owner with full diagnostics
|
||||
|
||||
### Rollback Awareness
|
||||
- After restarting a container, monitor for 5 minutes. If the same alert re-fires, do not attempt further restarts — report the persistent failure with logs.
|
||||
- Never delete data, drop volumes, or run `docker system prune --volumes`
|
||||
- `docker system prune -f` (without --volumes) is allowed for disk cleanup
|
||||
|
||||
### Command Logging
|
||||
- Report every command executed and its output to the Matrix room, either inline or as a summary after investigation
|
||||
|
||||
### Wiki Documentation
|
||||
- After taking any corrective action, Spirit updates the relevant Wiki.js page to document what happened, what was done, and when
|
||||
- Wiki.js GraphQL API: `http://192.168.86.100:3002/graphql` (internal, from container via SSH or direct HTTP)
|
||||
- Authentication: Bearer token (long-lived JWT API key, stored as env var `WIKIJS_API_KEY` in Spirit's compose)
|
||||
- Spirit appends incident entries to the relevant wiki page (e.g., infrastructure/alerting page ID 121, or service-specific pages)
|
||||
- Documentation format: timestamp, alert name, severity, root cause (if identified), actions taken, outcome
|
||||
- Key wiki pages: infrastructure/overview (118), service-access (119), maintenance (120), alerting (121)
|
||||
- Uses GraphQL mutations: `pages { update(id, content, ...) }` to append to existing pages
|
||||
|
||||
## Alert Response Behavior
|
||||
|
||||
When Spirit sees an alert from @alertbot:
|
||||
|
||||
1. **Parse** the alert: extract alert name, severity, instance, description
|
||||
2. **Investigate**: run relevant diagnostics (docker ps, logs, df, free, curl health endpoints)
|
||||
3. **Remediate**: take corrective action based on alert type (respecting guardrails)
|
||||
4. **Report**: post findings and actions taken back to #notifications
|
||||
5. **Document**: update the relevant Wiki.js page with an incident entry (timestamp, alert, actions taken, outcome)
|
||||
|
||||
### Remediation Procedures (All Alert Types)
|
||||
|
||||
| Alert | Severity | Investigation | Remediation |
|
||||
|-------|----------|--------------|-------------|
|
||||
| ContainerDown | warning | `docker ps -a`, `docker logs <container>` | `docker restart <container>` or `docker compose up -d` |
|
||||
| HighDiskUsage | critical | `df -h`, `docker system df` | `docker system prune -f`, clear old logs |
|
||||
| DiskSpaceWarning | warning | `df -h`, `docker system df` | Report findings, suggest cleanup |
|
||||
| HighMemoryUsage | warning | `free -m`, `docker stats --no-stream` | Identify memory hog, restart if needed |
|
||||
| CriticalMemoryUsage | critical | Same as above | Restart heaviest non-protected container, report |
|
||||
| HighCPUUsage | warning | `top -bn1`, `docker stats --no-stream` | Identify CPU hog, report |
|
||||
| CriticalCPUUsage | critical | Same as above | Identify and restart CPU hog if safe |
|
||||
| HighSystemLoad | warning | `uptime`, `top -bn1`, `docker stats` | Report findings, identify load source |
|
||||
| NFSMountDown | critical | `mount \| grep nfs`, `ping 192.168.86.30` | Attempt remount, report if NAS unreachable |
|
||||
| NASDiskSpaceLow | warning | `df -h /mnt/docker-data` | Report, identify large consumers |
|
||||
| SlowHTTPResponse | warning | `curl -w` timing, `docker logs` | Restart slow service |
|
||||
| FrigateHighMemory | warning | `docker stats frigate` | `docker restart frigate` |
|
||||
| ServiceHighMemory | warning | `docker stats`, identify container | Restart if safe |
|
||||
| ServerDown | critical | SSH attempt, ping | Report — cannot fix hardware |
|
||||
| ServerRebooted | info | Check uptime, verify services | Report, verify all containers are running |
|
||||
| HighSwapUsage | warning | `free -m`, `docker stats` | Identify swap contributor, restart |
|
||||
| CriticalSwapUsage | critical | Same as above | Restart heaviest swap user |
|
||||
| PrometheusTargetDown | warning | `curl` the target, `docker ps` | Restart target service if down |
|
||||
| Watchdog | info | N/A | Ignore (dead-man's switch, silenced in Alertmanager) |
|
||||
| UnifiDeviceDown | warning | Check Unifi controller | Report — network device issue |
|
||||
|
||||
### Resolved Alerts
|
||||
|
||||
When Spirit sees a "resolved" alert, acknowledge briefly (no investigation needed).
|
||||
|
||||
### Catch-All Rule
|
||||
|
||||
For any unrecognized alert type not in the table above: investigate and report findings, but do not take remediation action.
|
||||
|
||||
## What Does Not Change
|
||||
|
||||
- Spirit's Matrix account, access token, homeserver config
|
||||
- Spirit's gateway port (18795) and OpenClaw config structure
|
||||
- Other agents (margaret, clippy, bob, travis)
|
||||
- Alertbot continues posting alerts — Spirit is additive
|
||||
- Prometheus, Alertmanager, alert rules — all unchanged
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Build `spirit-ops:latest` image on hf-pdocker-01
|
||||
2. Generate SSH keypair, collect host keys, deploy public key to nodes
|
||||
3. Write workspace identity files (SOUL.md, IDENTITY.md, USER.md) to NAS
|
||||
4. Update compose file (image, volume mounts, memory limit)
|
||||
5. Redeploy Spirit container
|
||||
6. Invite Spirit to #notifications room (after new container is running)
|
||||
7. Verify: check SSH connectivity, docker socket access, Matrix room presence
|
||||
8. Test: send a test alert through Alertmanager, confirm Spirit investigates and responds
|
||||
Reference in New Issue
Block a user