jeremy
9943fd4410
Spirit now monitors #notifications Matrix room for Prometheus alerts, auto-investigates issues, takes corrective action (container restarts, disk cleanup, etc.), and documents incidents in Wiki.js. Changes: - Custom spirit-ops Docker image (Docker CLI + SSH + diagnostic tools) - Docker socket mount for local container management - SSH keys for remote access to hf-pdocker-02 and bart - Wiki.js API integration for incident documentation - Prometheus probe for Spirit gateway health - Safety guardrails: protected services, rate limiting, escalation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
192 lines
10 KiB
Markdown
192 lines
10 KiB
Markdown
# Spirit Infrastructure Ops Agent
|
|
|
|
## Overview
|
|
|
|
Repurpose the idle Spirit OpenClaw agent (`@spirit:hobbs.farm`) as an autonomous infrastructure operations agent. Spirit joins the `#notifications:hobbs.farm` Matrix room, monitors alerts posted by `@alertbot:hobbs.farm`, investigates issues, takes corrective action, and reports findings back to the room.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Prometheus → Alertmanager → @alertbot → #notifications room
|
|
↓
|
|
@spirit reads alert
|
|
↓
|
|
Investigate (docker/SSH)
|
|
↓
|
|
Remediate + report to room
|
|
```
|
|
|
|
Spirit has full autonomy to take corrective action, subject to the safety guardrails defined below.
|
|
|
|
## Cluster Context
|
|
|
|
Spirit must have complete knowledge of the infrastructure:
|
|
|
|
- **hf-pdocker-01** (192.168.86.192) — Swarm Manager, 23GB RAM. Runs: Frigate, Frigate-Notify, clip-service, Open WebUI, Home Assistant, ESPHome, Mosquitto, Matrix/Synapse, Guacamole, AI Agents, Traefik, AdGuard, Monitoring stack, Farm Manager (:8888), Farm Agent (:8889). Docker socket available locally.
|
|
- **hf-pdocker-02** (192.168.86.100) — 7.6GB RAM. Runs: Unifi, Gitea, Semaphore, Wiki.js, Vaultwarden, Actual Budget, MCPO, Trivy, SearXNG, Unstructured API, n8n, NetClaw, Farm Agent, Watchtower. Access via SSH.
|
|
- **bart** (192.168.86.167) — 15GB RAM. Runs: Jellyfin, Sonarr, Radarr, SABnzbd, NZBHydra2, Jellyseerr, OpenVAS, Onyx, Farm Agent, Watchtower. Access via SSH.
|
|
- **Synology NAS** (192.168.86.30) — NFS storage at `/mnt/docker-data` on all nodes.
|
|
|
|
## Components
|
|
|
|
### 1. Custom Docker Image: `spirit-ops:latest`
|
|
|
|
Build location: `/tmp/spirit-ops-build/` on hf-pdocker-01.
|
|
|
|
Extends `margaret-openclaw:latest` with operational tooling:
|
|
|
|
- `docker-cli` (Docker CLI, no daemon — uses mounted socket)
|
|
- `openssh-client` (SSH to remote nodes)
|
|
- `curl`, `jq` (API calls, JSON parsing)
|
|
- `procps`, `net-tools`, `iputils-ping`, `dnsutils` (system diagnostics)
|
|
|
|
The Dockerfile installs these packages on top of the existing OpenClaw + Matrix plugin base.
|
|
|
|
### 2. Compose Changes
|
|
|
|
In `compose/hf-pdocker-01/ai-agents/docker-compose.yml`, modify the `spirit` service:
|
|
|
|
- Change image from `margaret-openclaw:latest` to `spirit-ops:latest`
|
|
- Add volume mount: `/var/run/docker.sock:/var/run/docker.sock` (local Docker access)
|
|
- Add volume mount: `/mnt/infrastructure/spirit/ssh:/root/.ssh:ro` (SSH keys — mounted to `/root/.ssh` since container runs as root)
|
|
- Increase `mem_limit` and `memswap_limit` to `3G` (diagnostic commands + alert handling need headroom beyond the 1.5G Node.js allocation)
|
|
- Add environment variable `WIKIJS_API_KEY` (the long-lived JWT token for Wiki.js GraphQL API)
|
|
- All other settings (ports, network mode) remain unchanged.
|
|
|
|
Other agent services (margaret, clippy, bob, travis) are not modified.
|
|
|
|
### 3. SSH Key Setup
|
|
|
|
- Generate ED25519 keypair on hf-pdocker-01
|
|
- Store private key at `/mnt/infrastructure/spirit/ssh/id_ed25519` (NAS, mounted into container)
|
|
- Store public key at `/mnt/infrastructure/spirit/ssh/id_ed25519.pub`
|
|
- Create `/mnt/infrastructure/spirit/ssh/config` with:
|
|
- `IdentityFile /root/.ssh/id_ed25519` (explicit path)
|
|
- Pre-populated `known_hosts` file with actual host keys from all 3 nodes (no `StrictHostKeyChecking=no`)
|
|
- Deploy public key to `~glyptic/.ssh/authorized_keys` on hf-pdocker-02 and bart
|
|
- SSH user: `glyptic` (has sudo on all nodes)
|
|
|
|
### 4. Workspace Identity Files
|
|
|
|
Rewrite Spirit's workspace files on NAS at `/mnt/infrastructure/spirit/workspace/`:
|
|
|
|
**SOUL.md** — Core identity:
|
|
- Spirit is an infrastructure operations agent for a homelab cluster
|
|
- Full autonomy to investigate and remediate alerts, subject to safety guardrails
|
|
- Always reports actions and findings to the Matrix room
|
|
- Prioritizes service availability and data safety
|
|
|
|
**IDENTITY.md** — Detailed operational knowledge:
|
|
- Complete cluster architecture (nodes, IPs, services, ports)
|
|
- Compose file locations on NAS (`/mnt/docker-data/compose/{node}/{service}/`)
|
|
- All alert types and remediation procedures (see table below)
|
|
- Docker commands for local ops, SSH commands for remote ops
|
|
- NFS mount details and known issues (root_squash, database user requirements)
|
|
- Safety guardrails and protected services list
|
|
- Escalation behavior and rate limiting rules
|
|
|
|
**USER.md** — Owner context:
|
|
- Jeremy, homelab operator
|
|
- Prefers autonomous action with post-action reporting
|
|
- Full trust for corrective actions within guardrails
|
|
|
|
### 5. Matrix Room Membership
|
|
|
|
- Invite `@spirit:hobbs.farm` to `!jYmqxbzEsvRgKYvwMP:hobbs.farm` using Synapse admin API
|
|
- Spirit's OpenClaw Matrix plugin auto-accepts room invites
|
|
- Spirit begins monitoring all messages in the room
|
|
- Spirit responds to alert messages from `@alertbot:hobbs.farm`
|
|
- Spirit distinguishes alerts (HTML-formatted messages from @alertbot with severity/color coding) from normal conversation
|
|
|
|
## Safety Guardrails
|
|
|
|
### Protected Services (never restart without explicit owner request)
|
|
- Prometheus, Alertmanager, matrix-alertmanager-receiver (restarting breaks the alerting pipeline)
|
|
- Traefik (restarting drops all external access)
|
|
- Spirit's own container and other AI agents
|
|
- Database containers (*-db, guacamole-db, etc.)
|
|
|
|
### Rate Limiting
|
|
- **Per-container cooldown**: Do not restart the same container more than once per 30 minutes
|
|
- **Global budget**: Maximum 5 remediation actions per hour
|
|
- **Escalation**: After 2 failed remediation attempts for the same alert, stop trying and report to the owner with full diagnostics
|
|
|
|
### Rollback Awareness
|
|
- After restarting a container, monitor for 5 minutes. If the same alert re-fires, do not attempt further restarts — report the persistent failure with logs.
|
|
- Never delete data, drop volumes, or run `docker system prune --volumes`
|
|
- `docker system prune -f` (without --volumes) is allowed for disk cleanup
|
|
|
|
### Command Logging
|
|
- Report every command executed and its output to the Matrix room, either inline or as a summary after investigation
|
|
|
|
### Wiki Documentation
|
|
- After taking any corrective action, Spirit updates the relevant Wiki.js page to document what happened, what was done, and when
|
|
- Wiki.js GraphQL API: `http://192.168.86.100:3002/graphql` (internal, from container via SSH or direct HTTP)
|
|
- Authentication: Bearer token (long-lived JWT API key, stored as env var `WIKIJS_API_KEY` in Spirit's compose)
|
|
- Spirit appends incident entries to the relevant wiki page (e.g., infrastructure/alerting page ID 121, or service-specific pages)
|
|
- Documentation format: timestamp, alert name, severity, root cause (if identified), actions taken, outcome
|
|
- Key wiki pages: infrastructure/overview (118), service-access (119), maintenance (120), alerting (121)
|
|
- Uses GraphQL mutations: `pages { update(id, content, ...) }` to append to existing pages
|
|
|
|
## Alert Response Behavior
|
|
|
|
When Spirit sees an alert from @alertbot:
|
|
|
|
1. **Parse** the alert: extract alert name, severity, instance, description
|
|
2. **Investigate**: run relevant diagnostics (docker ps, logs, df, free, curl health endpoints)
|
|
3. **Remediate**: take corrective action based on alert type (respecting guardrails)
|
|
4. **Report**: post findings and actions taken back to #notifications
|
|
5. **Document**: update the relevant Wiki.js page with an incident entry (timestamp, alert, actions taken, outcome)
|
|
|
|
### Remediation Procedures (All Alert Types)
|
|
|
|
| Alert | Severity | Investigation | Remediation |
|
|
|-------|----------|--------------|-------------|
|
|
| ContainerDown | warning | `docker ps -a`, `docker logs <container>` | `docker restart <container>` or `docker compose up -d` |
|
|
| HighDiskUsage | critical | `df -h`, `docker system df` | `docker system prune -f`, clear old logs |
|
|
| DiskSpaceWarning | warning | `df -h`, `docker system df` | Report findings, suggest cleanup |
|
|
| HighMemoryUsage | warning | `free -m`, `docker stats --no-stream` | Identify memory hog, restart if needed |
|
|
| CriticalMemoryUsage | critical | Same as above | Restart heaviest non-protected container, report |
|
|
| HighCPUUsage | warning | `top -bn1`, `docker stats --no-stream` | Identify CPU hog, report |
|
|
| CriticalCPUUsage | critical | Same as above | Identify and restart CPU hog if safe |
|
|
| HighSystemLoad | warning | `uptime`, `top -bn1`, `docker stats` | Report findings, identify load source |
|
|
| NFSMountDown | critical | `mount \| grep nfs`, `ping 192.168.86.30` | Attempt remount, report if NAS unreachable |
|
|
| NASDiskSpaceLow | warning | `df -h /mnt/docker-data` | Report, identify large consumers |
|
|
| SlowHTTPResponse | warning | `curl -w` timing, `docker logs` | Restart slow service |
|
|
| FrigateHighMemory | warning | `docker stats frigate` | `docker restart frigate` |
|
|
| ServiceHighMemory | warning | `docker stats`, identify container | Restart if safe |
|
|
| ServerDown | critical | SSH attempt, ping | Report — cannot fix hardware |
|
|
| ServerRebooted | info | Check uptime, verify services | Report, verify all containers are running |
|
|
| HighSwapUsage | warning | `free -m`, `docker stats` | Identify swap contributor, restart |
|
|
| CriticalSwapUsage | critical | Same as above | Restart heaviest swap user |
|
|
| PrometheusTargetDown | warning | `curl` the target, `docker ps` | Restart target service if down |
|
|
| Watchdog | info | N/A | Ignore (dead-man's switch, silenced in Alertmanager) |
|
|
| UnifiDeviceDown | warning | Check Unifi controller | Report — network device issue |
|
|
|
|
### Resolved Alerts
|
|
|
|
When Spirit sees a "resolved" alert, acknowledge briefly (no investigation needed).
|
|
|
|
### Catch-All Rule
|
|
|
|
For any unrecognized alert type not in the table above: investigate and report findings, but do not take remediation action.
|
|
|
|
## What Does Not Change
|
|
|
|
- Spirit's Matrix account, access token, homeserver config
|
|
- Spirit's gateway port (18795) and OpenClaw config structure
|
|
- Other agents (margaret, clippy, bob, travis)
|
|
- Alertbot continues posting alerts — Spirit is additive
|
|
- Prometheus, Alertmanager, alert rules — all unchanged
|
|
|
|
## Implementation Steps
|
|
|
|
1. Build `spirit-ops:latest` image on hf-pdocker-01
|
|
2. Generate SSH keypair, collect host keys, deploy public key to nodes
|
|
3. Write workspace identity files (SOUL.md, IDENTITY.md, USER.md) to NAS
|
|
4. Update compose file (image, volume mounts, memory limit)
|
|
5. Redeploy Spirit container
|
|
6. Invite Spirit to #notifications room (after new container is running)
|
|
7. Verify: check SSH connectivity, docker socket access, Matrix room presence
|
|
8. Test: send a test alert through Alertmanager, confirm Spirit investigates and responds
|