hobbs-iac/docs/plans/2026-03-05-farm-manager.md

# Farm Manager Implementation Plan

> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task.

**Goal:** Build a web dashboard to manage Docker services across 3 homelab nodes with full control and service grouping.

**Architecture:** Node agents on each Docker host expose REST APIs over the local Docker socket. A central API server proxies requests and manages groups. Static HTML/CSS/JS frontend served by the API server.

**Tech Stack:** Python 3.12, FastAPI, docker-py (aiodocker), httpx, pytest, vanilla HTML/CSS/JS

**Repo:** git.hobbs.farm/jeremy/farm-manager (Gitea issues #1-#16 track backlog)

**Development:** Code is developed locally at `/mnt/c/Users/Jeremy/projects/farm-manager/`, pushed to Gitea, then deployed to nodes.

---

### Task 1: Project Scaffold + Shared Models

**Gitea issues:** Prerequisite for all

**Files:**
- Create: `agent/__init__.py`
- Create: `agent/models.py`
- Create: `agent/requirements.txt`
- Create: `server/__init__.py`
- Create: `server/models.py`
- Create: `server/requirements.txt`
- Create: `tests/__init__.py`
- Create: `tests/conftest.py`
- Create: `config.json`
- Create: `groups.json`
- Create: `.gitignore`

**Step 1: Clone repo and create project structure**

```bash
cd /mnt/c/Users/Jeremy/projects
git clone http://192.168.86.100:3003/jeremy/farm-manager.git
cd farm-manager
mkdir -p agent server/routes server/static tests
touch agent/__init__.py server/__init__.py server/routes/__init__.py tests/__init__.py
```

**Step 2: Create shared pydantic models for the agent**

`agent/models.py`:
```python
from pydantic import BaseModel


class ContainerInfo(BaseModel):
    id: str
    name: str
    status: str  # running, exited, paused, restarting
    image: str
    created: str
    uptime: str | None = None
    is_swarm: bool = False
    swarm_service: str | None = None


class ContainerAction(BaseModel):
    success: bool
    message: str


class LogsResponse(BaseModel):
    container: str
    logs: str


class HealthResponse(BaseModel):
    status: str
    hostname: str
    containers_total: int
```

**Step 3: Create shared pydantic models for the server**

`server/models.py`:
```python
from pydantic import BaseModel


class NodeConfig(BaseModel):
    name: str
    host: str
    agent_port: int = 8889


class NodeStatus(BaseModel):
    name: str
    host: str
    healthy: bool
    containers_total: int = 0
    error: str | None = None


class ServiceRef(BaseModel):
    node: str
    container: str


class Group(BaseModel):
    id: str
    name: str
    services: list[ServiceRef] = []


class GroupsData(BaseModel):
    groups: list[Group] = []


class ServiceInfo(BaseModel):
    id: str
    name: str
    node: str
    status: str
    image: str
    created: str
    uptime: str | None = None
    is_swarm: bool = False
    swarm_service: str | None = None
```

**Step 4: Create requirements files**

`agent/requirements.txt`:
```
fastapi>=0.115.0
uvicorn>=0.34.0
docker>=7.0.0
```

`server/requirements.txt`:
```
fastapi>=0.115.0
uvicorn>=0.34.0
httpx>=0.28.0
aiofiles>=24.1.0
```

`tests/requirements.txt`:
```
pytest>=8.0.0
pytest-asyncio>=0.24.0
httpx>=0.28.0
```

**Step 5: Create config files**

`config.json`:
```json
{
  "nodes": [
    {"name": "hf-pdocker-01", "host": "192.168.86.192", "agent_port": 8889},
    {"name": "hf-pdocker-02", "host": "192.168.86.100", "agent_port": 8889},
    {"name": "bart", "host": "192.168.86.167", "agent_port": 8889}
  ]
}
```

`groups.json`:
```json
{
  "groups": [
    {
      "id": "ai-agents",
      "name": "AI Agents",
      "services": [
        {"node": "hf-pdocker-01", "container": "ai-agents_margaret"},
        {"node": "hf-pdocker-01", "container": "ai-agents_spirit"},
        {"node": "hf-pdocker-01", "container": "ai-agents_bob"},
        {"node": "hf-pdocker-01", "container": "ai-agents_clippy"}
      ]
    },
    {
      "id": "smart-home",
      "name": "Smart Home",
      "services": [
        {"node": "hf-pdocker-01", "container": "smarthome_homeassistant"},
        {"node": "hf-pdocker-01", "container": "smarthome_esphome"},
        {"node": "hf-pdocker-01", "container": "smarthome_mosquitto"}
      ]
    },
    {
      "id": "monitoring",
      "name": "Monitoring",
      "services": [
        {"node": "hf-pdocker-01", "container": "monitoring_grafana"},
        {"node": "hf-pdocker-01", "container": "monitoring_prometheus"},
        {"node": "hf-pdocker-01", "container": "monitoring_alertmanager"},
        {"node": "hf-pdocker-01", "container": "monitoring_influxdb"},
        {"node": "hf-pdocker-01", "container": "monitoring_blackbox-exporter"}
      ]
    },
    {
      "id": "media",
      "name": "Media",
      "services": [
        {"node": "bart", "container": "jellyfin"},
        {"node": "bart", "container": "sonarr"},
        {"node": "bart", "container": "radarr"},
        {"node": "bart", "container": "sabnzbd"},
        {"node": "bart", "container": "nzbhydra2"},
        {"node": "bart", "container": "jellyseerr"}
      ]
    },
    {
      "id": "matrix",
      "name": "Matrix",
      "services": [
        {"node": "hf-pdocker-01", "container": "matrix_synapse"},
        {"node": "hf-pdocker-01", "container": "matrix_synapse-db"},
        {"node": "hf-pdocker-01", "container": "matrix_wellknown"}
      ]
    },
    {
      "id": "guacamole",
      "name": "Guacamole",
      "services": [
        {"node": "hf-pdocker-01", "container": "guacamole_guacamole"},
        {"node": "hf-pdocker-01", "container": "guacamole_guacd"},
        {"node": "hf-pdocker-01", "container": "guacamole_guacamole-db"}
      ]
    },
    {
      "id": "dev-tools",
      "name": "Dev Tools",
      "services": [
        {"node": "hf-pdocker-02", "container": "gitea"},
        {"node": "hf-pdocker-02", "container": "gitea-db"},
        {"node": "hf-pdocker-02", "container": "semaphore"},
        {"node": "hf-pdocker-02", "container": "semaphore-db"},
        {"node": "hf-pdocker-02", "container": "wikijs"},
        {"node": "hf-pdocker-02", "container": "wikijs-db"}
      ]
    },
    {
      "id": "security",
      "name": "Security",
      "services": [
        {"node": "hf-pdocker-02", "container": "vaultwarden"},
        {"node": "hf-pdocker-02", "container": "trivy"},
        {"node": "bart", "container": "openvas"}
      ]
    },
    {
      "id": "network",
      "name": "Network",
      "services": [
        {"node": "hf-pdocker-01", "container": "traefik_traefik"},
        {"node": "hf-pdocker-01", "container": "adguard_adguardhome"},
        {"node": "hf-pdocker-02", "container": "unifi"},
        {"node": "hf-pdocker-02", "container": "unifi-db"}
      ]
    },
    {
      "id": "frigate",
      "name": "Frigate",
      "services": [
        {"node": "hf-pdocker-01", "container": "frigate"},
        {"node": "hf-pdocker-01", "container": "frigate-notify"}
      ]
    }
  ]
}
```

`.gitignore`:
```
__pycache__/
*.pyc
.pytest_cache/
*.egg-info/
venv/
.env
```

**Step 6: Create test conftest**

`tests/conftest.py`:
```python
import pytest


@pytest.fixture
def sample_container():
    return {
        "id": "abc123",
        "name": "test-container",
        "status": "running",
        "image": "nginx:latest",
        "created": "2026-03-05T00:00:00Z",
        "uptime": "2 hours",
        "is_swarm": False,
        "swarm_service": None,
    }


@pytest.fixture
def sample_group():
    return {
        "id": "test-group",
        "name": "Test Group",
        "services": [
            {"node": "hf-pdocker-01", "container": "test-container"},
        ],
    }
```

**Step 7: Commit scaffold**

```bash
git add -A
git commit -m "feat: project scaffold with models, config, and test fixtures"
git push origin main
```

---

### Task 2: Node Agent — Docker Operations (TDD)

**Gitea issues:** #1, #12

**Files:**
- Create: `tests/test_agent_docker_ops.py`
- Create: `agent/docker_ops.py`

**Step 1: Write failing tests for Docker operations**

`tests/test_agent_docker_ops.py`:
```python
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from agent.docker_ops import DockerOps


@pytest.fixture
def mock_docker_client():
    client = MagicMock()
    client.containers = MagicMock()
    return client


@pytest.fixture
def docker_ops(mock_docker_client):
    with patch("agent.docker_ops.docker.DockerClient", return_value=mock_docker_client):
        ops = DockerOps()
        ops.client = mock_docker_client
        return ops


class TestListContainers:
    def test_list_returns_container_info(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.id = "abc123def456"
        mock_container.name = "test-app"
        mock_container.status = "running"
        mock_container.image.tags = ["nginx:latest"]
        mock_container.attrs = {
            "Created": "2026-03-05T00:00:00Z",
            "State": {"StartedAt": "2026-03-05T01:00:00Z"},
        }
        mock_container.labels = {}
        mock_docker_client.containers.list.return_value = [mock_container]

        result = docker_ops.list_containers()

        assert len(result) == 1
        assert result[0]["name"] == "test-app"
        assert result[0]["status"] == "running"
        assert result[0]["image"] == "nginx:latest"
        assert result[0]["is_swarm"] is False

    def test_list_detects_swarm_containers(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.id = "abc123"
        mock_container.name = "stack_service.1.taskid"
        mock_container.status = "running"
        mock_container.image.tags = ["app:latest"]
        mock_container.attrs = {
            "Created": "2026-03-05T00:00:00Z",
            "State": {"StartedAt": "2026-03-05T01:00:00Z"},
        }
        mock_container.labels = {
            "com.docker.swarm.service.name": "stack_service",
        }
        mock_docker_client.containers.list.return_value = [mock_container]

        result = docker_ops.list_containers()

        assert result[0]["is_swarm"] is True
        assert result[0]["swarm_service"] == "stack_service"

    def test_list_includes_stopped_containers(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.id = "abc123"
        mock_container.name = "stopped-app"
        mock_container.status = "exited"
        mock_container.image.tags = ["app:latest"]
        mock_container.attrs = {
            "Created": "2026-03-05T00:00:00Z",
            "State": {"StartedAt": "2026-03-05T01:00:00Z"},
        }
        mock_container.labels = {}
        mock_docker_client.containers.list.return_value = [mock_container]

        result = docker_ops.list_containers()

        assert result[0]["status"] == "exited"


class TestContainerActions:
    def test_start_container(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.name = "test-app"
        mock_docker_client.containers.get.return_value = mock_container

        result = docker_ops.start_container("abc123")

        mock_container.start.assert_called_once()
        assert result["success"] is True

    def test_stop_container(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.name = "test-app"
        mock_docker_client.containers.get.return_value = mock_container

        result = docker_ops.stop_container("abc123")

        mock_container.stop.assert_called_once()
        assert result["success"] is True

    def test_restart_container(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.name = "test-app"
        mock_docker_client.containers.get.return_value = mock_container

        result = docker_ops.restart_container("abc123")

        mock_container.restart.assert_called_once()
        assert result["success"] is True

    def test_start_nonexistent_container(self, docker_ops, mock_docker_client):
        import docker.errors
        mock_docker_client.containers.get.side_effect = docker.errors.NotFound("not found")

        result = docker_ops.start_container("doesntexist")

        assert result["success"] is False
        assert "not found" in result["message"].lower()


class TestContainerLogs:
    def test_get_logs(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.name = "test-app"
        mock_container.logs.return_value = b"line1\nline2\nline3"
        mock_docker_client.containers.get.return_value = mock_container

        result = docker_ops.get_logs("abc123", tail=100)

        assert result["container"] == "test-app"
        assert "line1" in result["logs"]


class TestPullImage:
    def test_pull_image(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.name = "test-app"
        mock_container.image.tags = ["nginx:latest"]
        mock_docker_client.containers.get.return_value = mock_container
        mock_docker_client.images.pull.return_value = MagicMock()

        result = docker_ops.pull_image("abc123")

        mock_docker_client.images.pull.assert_called_once_with("nginx:latest")
        assert result["success"] is True


class TestSwarmActions:
    def test_stop_swarm_service_scales_to_zero(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.name = "stack_svc.1.taskid"
        mock_container.labels = {"com.docker.swarm.service.name": "stack_svc"}
        mock_docker_client.containers.get.return_value = mock_container

        mock_service = MagicMock()
        mock_docker_client.services.get.return_value = mock_service

        result = docker_ops.stop_container("abc123")

        mock_service.scale.assert_called_once_with(0)
        assert result["success"] is True

    def test_start_swarm_service_scales_to_one(self, docker_ops, mock_docker_client):
        mock_container = MagicMock()
        mock_container.name = "stack_svc.1.taskid"
        mock_container.labels = {"com.docker.swarm.service.name": "stack_svc"}
        mock_docker_client.containers.get.return_value = mock_container

        mock_service = MagicMock()
        mock_docker_client.services.get.return_value = mock_service

        result = docker_ops.start_container("abc123")

        mock_service.scale.assert_called_once_with(1)
        assert result["success"] is True
```

**Step 2: Run tests — verify they fail**

```bash
cd /mnt/c/Users/Jeremy/projects/farm-manager
pip install -r agent/requirements.txt -r tests/requirements.txt
pytest tests/test_agent_docker_ops.py -v
```
Expected: FAIL — `ModuleNotFoundError: No module named 'agent.docker_ops'`

**Step 3: Implement DockerOps**

`agent/docker_ops.py`:
```python
import socket
from datetime import datetime, timezone

import docker
import docker.errors


class DockerOps:
    def __init__(self):
        self.client = docker.DockerClient.from_env()
        self.hostname = socket.gethostname()

    def _is_swarm(self, container) -> bool:
        return "com.docker.swarm.service.name" in container.labels

    def _get_swarm_service_name(self, container) -> str | None:
        return container.labels.get("com.docker.swarm.service.name")

    def _format_uptime(self, started_at: str) -> str:
        try:
            start = datetime.fromisoformat(started_at.replace("Z", "+00:00"))
            delta = datetime.now(timezone.utc) - start
            hours, remainder = divmod(int(delta.total_seconds()), 3600)
            minutes, _ = divmod(remainder, 60)
            if hours >= 24:
                days = hours // 24
                hours = hours % 24
                return f"{days}d {hours}h {minutes}m"
            return f"{hours}h {minutes}m"
        except Exception:
            return None

    def list_containers(self) -> list[dict]:
        containers = self.client.containers.list(all=True)
        result = []
        for c in containers:
            tags = c.image.tags
            image = tags[0] if tags else "unknown"
            started_at = c.attrs.get("State", {}).get("StartedAt", "")
            is_swarm = self._is_swarm(c)
            result.append({
                "id": c.id[:12],
                "name": c.name,
                "status": c.status,
                "image": image,
                "created": c.attrs.get("Created", ""),
                "uptime": self._format_uptime(started_at) if c.status == "running" else None,
                "is_swarm": is_swarm,
                "swarm_service": self._get_swarm_service_name(c) if is_swarm else None,
            })
        return result

    def start_container(self, container_id: str) -> dict:
        try:
            container = self.client.containers.get(container_id)
            if self._is_swarm(container):
                svc_name = self._get_swarm_service_name(container)
                service = self.client.services.get(svc_name)
                service.scale(1)
                return {"success": True, "message": f"Scaled {svc_name} to 1"}
            container.start()
            return {"success": True, "message": f"Started {container.name}"}
        except docker.errors.NotFound:
            return {"success": False, "message": "Container not found"}
        except Exception as e:
            return {"success": False, "message": str(e)}

    def stop_container(self, container_id: str) -> dict:
        try:
            container = self.client.containers.get(container_id)
            if self._is_swarm(container):
                svc_name = self._get_swarm_service_name(container)
                service = self.client.services.get(svc_name)
                service.scale(0)
                return {"success": True, "message": f"Scaled {svc_name} to 0"}
            container.stop()
            return {"success": True, "message": f"Stopped {container.name}"}
        except docker.errors.NotFound:
            return {"success": False, "message": "Container not found"}
        except Exception as e:
            return {"success": False, "message": str(e)}

    def restart_container(self, container_id: str) -> dict:
        try:
            container = self.client.containers.get(container_id)
            if self._is_swarm(container):
                svc_name = self._get_swarm_service_name(container)
                service = self.client.services.get(svc_name)
                service.force_update()
                return {"success": True, "message": f"Force-updated {svc_name}"}
            container.restart()
            return {"success": True, "message": f"Restarted {container.name}"}
        except docker.errors.NotFound:
            return {"success": False, "message": "Container not found"}
        except Exception as e:
            return {"success": False, "message": str(e)}

    def get_logs(self, container_id: str, tail: int = 100) -> dict:
        try:
            container = self.client.containers.get(container_id)
            logs = container.logs(tail=tail, timestamps=True).decode("utf-8", errors="replace")
            return {"container": container.name, "logs": logs}
        except docker.errors.NotFound:
            return {"container": container_id, "logs": "Container not found"}
        except Exception as e:
            return {"container": container_id, "logs": str(e)}

    def pull_image(self, container_id: str) -> dict:
        try:
            container = self.client.containers.get(container_id)
            tags = container.image.tags
            if not tags:
                return {"success": False, "message": "No image tag found"}
            image_name = tags[0]
            self.client.images.pull(image_name)
            return {"success": True, "message": f"Pulled {image_name}"}
        except docker.errors.NotFound:
            return {"success": False, "message": "Container not found"}
        except Exception as e:
            return {"success": False, "message": str(e)}

    def get_health(self) -> dict:
        containers = self.client.containers.list(all=True)
        return {
            "status": "ok",
            "hostname": self.hostname,
            "containers_total": len(containers),
        }
```

**Step 4: Run tests — verify they pass**

```bash
pytest tests/test_agent_docker_ops.py -v
```
Expected: All PASS

**Step 5: Commit**

```bash
git add agent/docker_ops.py tests/test_agent_docker_ops.py
git commit -m "feat(agent): Docker operations with Swarm/Compose detection (TDD)

Closes #1, progresses #12"
git push origin main
```

---

### Task 3: Node Agent — API Endpoints (TDD)

**Gitea issues:** #1

**Files:**
- Create: `tests/test_agent_api.py`
- Create: `agent/main.py`

**Step 1: Write failing tests for agent API**

`tests/test_agent_api.py`:
```python
import pytest
from unittest.mock import patch, MagicMock
from fastapi.testclient import TestClient


@pytest.fixture
def mock_ops():
    with patch("agent.main.ops") as mock:
        yield mock


@pytest.fixture
def client(mock_ops):
    from agent.main import app
    return TestClient(app)


class TestHealthEndpoint:
    def test_health_returns_ok(self, client, mock_ops):
        mock_ops.get_health.return_value = {
            "status": "ok",
            "hostname": "test-node",
            "containers_total": 5,
        }
        resp = client.get("/health")
        assert resp.status_code == 200
        assert resp.json()["status"] == "ok"


class TestListContainers:
    def test_list_containers(self, client, mock_ops):
        mock_ops.list_containers.return_value = [
            {
                "id": "abc123",
                "name": "test-app",
                "status": "running",
                "image": "nginx:latest",
                "created": "2026-03-05T00:00:00Z",
                "uptime": "2h 30m",
                "is_swarm": False,
                "swarm_service": None,
            }
        ]
        resp = client.get("/containers")
        assert resp.status_code == 200
        data = resp.json()
        assert len(data) == 1
        assert data[0]["name"] == "test-app"


class TestContainerActions:
    def test_start_container(self, client, mock_ops):
        mock_ops.start_container.return_value = {"success": True, "message": "Started"}
        resp = client.post("/containers/abc123/start")
        assert resp.status_code == 200
        assert resp.json()["success"] is True

    def test_stop_container(self, client, mock_ops):
        mock_ops.stop_container.return_value = {"success": True, "message": "Stopped"}
        resp = client.post("/containers/abc123/stop")
        assert resp.status_code == 200
        assert resp.json()["success"] is True

    def test_restart_container(self, client, mock_ops):
        mock_ops.restart_container.return_value = {"success": True, "message": "Restarted"}
        resp = client.post("/containers/abc123/restart")
        assert resp.status_code == 200
        assert resp.json()["success"] is True


class TestLogs:
    def test_get_logs(self, client, mock_ops):
        mock_ops.get_logs.return_value = {"container": "test-app", "logs": "log line 1\nlog line 2"}
        resp = client.get("/containers/abc123/logs?tail=50")
        assert resp.status_code == 200
        assert "log line 1" in resp.json()["logs"]

    def test_get_logs_default_tail(self, client, mock_ops):
        mock_ops.get_logs.return_value = {"container": "test-app", "logs": "logs"}
        resp = client.get("/containers/abc123/logs")
        mock_ops.get_logs.assert_called_once_with("abc123", tail=200)


class TestPull:
    def test_pull_image(self, client, mock_ops):
        mock_ops.pull_image.return_value = {"success": True, "message": "Pulled nginx:latest"}
        resp = client.post("/containers/abc123/pull")
        assert resp.status_code == 200
        assert resp.json()["success"] is True
```

**Step 2: Run tests — verify they fail**

```bash
pytest tests/test_agent_api.py -v
```
Expected: FAIL — `ModuleNotFoundError: No module named 'agent.main'`

**Step 3: Implement agent FastAPI app**

`agent/main.py`:
```python
from fastapi import FastAPI, Query

from agent.docker_ops import DockerOps

app = FastAPI(title="Farm Manager Agent")
ops = DockerOps()


@app.get("/health")
def health():
    return ops.get_health()


@app.get("/containers")
def list_containers():
    return ops.list_containers()


@app.post("/containers/{container_id}/start")
def start_container(container_id: str):
    return ops.start_container(container_id)


@app.post("/containers/{container_id}/stop")
def stop_container(container_id: str):
    return ops.stop_container(container_id)


@app.post("/containers/{container_id}/restart")
def restart_container(container_id: str):
    return ops.restart_container(container_id)


@app.get("/containers/{container_id}/logs")
def get_logs(container_id: str, tail: int = Query(default=200)):
    return ops.get_logs(container_id, tail=tail)


@app.post("/containers/{container_id}/pull")
def pull_image(container_id: str):
    return ops.pull_image(container_id)


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8889)
```

**Step 4: Run tests — verify they pass**

```bash
pytest tests/test_agent_api.py -v
```
Expected: All PASS

**Step 5: Commit**

```bash
git add agent/main.py tests/test_agent_api.py
git commit -m "feat(agent): FastAPI endpoints for container management (TDD)

Closes #1"
git push origin main
```

---

### Task 4: Node Agent — Dockerfile

**Gitea issues:** #2

**Files:**
- Create: `agent/Dockerfile`

**Step 1: Create Dockerfile**

`agent/Dockerfile`:
```dockerfile
FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8889

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8889"]
```

**Step 2: Commit**

```bash
git add agent/Dockerfile
git commit -m "feat(agent): Dockerfile for node agent

Closes #2"
git push origin main
```

---

### Task 5: API Server — Service Proxy (TDD)

**Gitea issues:** #3, #13

**Files:**
- Create: `tests/test_server_proxy.py`
- Create: `server/main.py`
- Create: `server/routes/services.py`
- Create: `server/routes/nodes.py`

**Step 1: Write failing tests for server proxy**

`tests/test_server_proxy.py`:
```python
import json
import pytest
from unittest.mock import AsyncMock, patch, MagicMock
from fastapi.testclient import TestClient


@pytest.fixture
def mock_httpx():
    with patch("server.routes.services.httpx.AsyncClient") as mock_cls:
        mock_client = AsyncMock()
        mock_cls.return_value.__aenter__ = AsyncMock(return_value=mock_client)
        mock_cls.return_value.__aexit__ = AsyncMock(return_value=False)
        yield mock_client


@pytest.fixture
def client():
    from server.main import app
    return TestClient(app)


class TestListNodes:
    def test_list_nodes(self, client):
        with patch("server.routes.nodes.httpx.AsyncClient") as mock_cls:
            mock_client = AsyncMock()
            mock_cls.return_value.__aenter__ = AsyncMock(return_value=mock_client)
            mock_cls.return_value.__aexit__ = AsyncMock(return_value=False)

            mock_resp = AsyncMock()
            mock_resp.status_code = 200
            mock_resp.json.return_value = {"status": "ok", "hostname": "hf-pdocker-01", "containers_total": 10}
            mock_client.get.return_value = mock_resp

            resp = client.get("/api/nodes")
            assert resp.status_code == 200
            data = resp.json()
            assert len(data) == 3
            assert data[0]["name"] == "hf-pdocker-01"


class TestListServices:
    def test_list_all_services(self, client):
        with patch("server.routes.services.httpx.AsyncClient") as mock_cls:
            mock_client = AsyncMock()
            mock_cls.return_value.__aenter__ = AsyncMock(return_value=mock_client)
            mock_cls.return_value.__aexit__ = AsyncMock(return_value=False)

            mock_resp = AsyncMock()
            mock_resp.status_code = 200
            mock_resp.json.return_value = [
                {"id": "abc", "name": "nginx", "status": "running", "image": "nginx:latest",
                 "created": "", "uptime": "1h", "is_swarm": False, "swarm_service": None}
            ]
            mock_client.get.return_value = mock_resp

            resp = client.get("/api/services")
            assert resp.status_code == 200
            data = resp.json()
            assert len(data) >= 1
            assert data[0]["node"] in ["hf-pdocker-01", "hf-pdocker-02", "bart"]


class TestServiceActions:
    def test_start_service(self, client):
        with patch("server.routes.services.httpx.AsyncClient") as mock_cls:
            mock_client = AsyncMock()
            mock_cls.return_value.__aenter__ = AsyncMock(return_value=mock_client)
            mock_cls.return_value.__aexit__ = AsyncMock(return_value=False)

            mock_resp = AsyncMock()
            mock_resp.status_code = 200
            mock_resp.json.return_value = {"success": True, "message": "Started"}
            mock_client.post.return_value = mock_resp

            resp = client.post("/api/services/hf-pdocker-01/abc123/start")
            assert resp.status_code == 200
            assert resp.json()["success"] is True

    def test_invalid_node_returns_404(self, client):
        resp = client.post("/api/services/nonexistent-node/abc123/start")
        assert resp.status_code == 404
```

**Step 2: Run tests — verify they fail**

```bash
pytest tests/test_server_proxy.py -v
```
Expected: FAIL

**Step 3: Implement server routes**

`server/routes/nodes.py`:
```python
import httpx
from fastapi import APIRouter

from server.config import get_nodes

router = APIRouter()


@router.get("/api/nodes")
async def list_nodes():
    nodes = get_nodes()
    results = []
    for node in nodes:
        url = f"http://{node['host']}:{node['agent_port']}/health"
        try:
            async with httpx.AsyncClient(timeout=5.0) as client:
                resp = await client.get(url)
                if resp.status_code == 200:
                    health = resp.json()
                    results.append({
                        "name": node["name"],
                        "host": node["host"],
                        "healthy": True,
                        "containers_total": health.get("containers_total", 0),
                    })
                else:
                    results.append({"name": node["name"], "host": node["host"], "healthy": False, "error": f"HTTP {resp.status_code}"})
        except Exception as e:
            results.append({"name": node["name"], "host": node["host"], "healthy": False, "error": str(e)})
    return results
```

`server/routes/services.py`:
```python
import httpx
from fastapi import APIRouter, HTTPException, Query

from server.config import get_node_url

router = APIRouter()


@router.get("/api/services")
async def list_services():
    from server.config import get_nodes
    all_services = []
    for node in get_nodes():
        url = f"http://{node['host']}:{node['agent_port']}/containers"
        try:
            async with httpx.AsyncClient(timeout=10.0) as client:
                resp = await client.get(url)
                if resp.status_code == 200:
                    containers = resp.json()
                    for c in containers:
                        c["node"] = node["name"]
                    all_services.extend(containers)
        except Exception:
            pass
    return all_services


@router.post("/api/services/{node}/{container_id}/start")
async def start_service(node: str, container_id: str):
    base = get_node_url(node)
    if not base:
        raise HTTPException(status_code=404, detail=f"Node {node} not found")
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(f"{base}/containers/{container_id}/start")
        return resp.json()


@router.post("/api/services/{node}/{container_id}/stop")
async def stop_service(node: str, container_id: str):
    base = get_node_url(node)
    if not base:
        raise HTTPException(status_code=404, detail=f"Node {node} not found")
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(f"{base}/containers/{container_id}/stop")
        return resp.json()


@router.post("/api/services/{node}/{container_id}/restart")
async def restart_service(node: str, container_id: str):
    base = get_node_url(node)
    if not base:
        raise HTTPException(status_code=404, detail=f"Node {node} not found")
    async with httpx.AsyncClient(timeout=30.0) as client:
        resp = await client.post(f"{base}/containers/{container_id}/restart")
        return resp.json()


@router.get("/api/services/{node}/{container_id}/logs")
async def get_logs(node: str, container_id: str, tail: int = Query(default=200)):
    base = get_node_url(node)
    if not base:
        raise HTTPException(status_code=404, detail=f"Node {node} not found")
    async with httpx.AsyncClient(timeout=10.0) as client:
        resp = await client.get(f"{base}/containers/{container_id}/logs", params={"tail": tail})
        return resp.json()


@router.post("/api/services/{node}/{container_id}/pull")
async def pull_image(node: str, container_id: str):
    base = get_node_url(node)
    if not base:
        raise HTTPException(status_code=404, detail=f"Node {node} not found")
    async with httpx.AsyncClient(timeout=120.0) as client:
        resp = await client.post(f"{base}/containers/{container_id}/pull")
        return resp.json()
```

`server/config.py`:
```python
import json
import os

_config = None


def _load_config():
    global _config
    if _config is None:
        config_path = os.environ.get("CONFIG_PATH", "/app/config.json")
        with open(config_path) as f:
            _config = json.load(f)
    return _config


def get_nodes() -> list[dict]:
    return _load_config()["nodes"]


def get_node_url(node_name: str) -> str | None:
    for node in get_nodes():
        if node["name"] == node_name:
            return f"http://{node['host']}:{node['agent_port']}"
    return None
```

`server/main.py`:
```python
import os

from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles

from server.routes.nodes import router as nodes_router
from server.routes.services import router as services_router

app = FastAPI(title="Farm Manager")

app.include_router(nodes_router)
app.include_router(services_router)


@app.get("/health")
def health():
    return {"status": "ok"}


static_dir = os.path.join(os.path.dirname(__file__), "static")
if os.path.isdir(static_dir):
    app.mount("/", StaticFiles(directory=static_dir, html=True), name="static")


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8888)
```

**Step 4: Run tests — verify they pass**

```bash
CONFIG_PATH=config.json pytest tests/test_server_proxy.py -v
```
Expected: All PASS

**Step 5: Commit**

```bash
git add server/ tests/test_server_proxy.py
git commit -m "feat(server): API proxy routes for services and nodes (TDD)

Closes #3, progresses #13"
git push origin main
```

---

### Task 6: API Server — Group CRUD (TDD)

**Gitea issues:** #4, #13

**Files:**
- Create: `tests/test_server_groups.py`
- Create: `server/routes/groups.py`
- Create: `server/group_store.py`

**Step 1: Write failing tests**

`tests/test_server_groups.py`:
```python
import json
import os
import tempfile
import pytest
from fastapi.testclient import TestClient


@pytest.fixture
def groups_file():
    f = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False)
    json.dump({"groups": []}, f)
    f.close()
    yield f.name
    os.unlink(f.name)


@pytest.fixture
def client(groups_file):
    os.environ["GROUPS_PATH"] = groups_file
    os.environ["CONFIG_PATH"] = "config.json"
    from server.group_store import GroupStore
    GroupStore._instance = None
    from server.main import app
    return TestClient(app)


class TestGroupCRUD:
    def test_list_groups_empty(self, client):
        resp = client.get("/api/groups")
        assert resp.status_code == 200
        assert resp.json() == []

    def test_create_group(self, client):
        resp = client.post("/api/groups", json={
            "id": "test-group",
            "name": "Test Group",
            "services": [{"node": "hf-pdocker-01", "container": "test"}],
        })
        assert resp.status_code == 200
        assert resp.json()["id"] == "test-group"

    def test_create_group_then_list(self, client):
        client.post("/api/groups", json={
            "id": "grp1", "name": "Group 1", "services": [],
        })
        resp = client.get("/api/groups")
        assert len(resp.json()) == 1
        assert resp.json()[0]["name"] == "Group 1"

    def test_update_group(self, client):
        client.post("/api/groups", json={
            "id": "grp1", "name": "Group 1", "services": [],
        })
        resp = client.put("/api/groups/grp1", json={
            "id": "grp1", "name": "Updated Group", "services": [{"node": "bart", "container": "jellyfin"}],
        })
        assert resp.status_code == 200
        assert resp.json()["name"] == "Updated Group"
        assert len(resp.json()["services"]) == 1

    def test_delete_group(self, client):
        client.post("/api/groups", json={
            "id": "grp1", "name": "Group 1", "services": [],
        })
        resp = client.delete("/api/groups/grp1")
        assert resp.status_code == 200
        assert len(client.get("/api/groups").json()) == 0

    def test_update_nonexistent_returns_404(self, client):
        resp = client.put("/api/groups/nope", json={
            "id": "nope", "name": "Nope", "services": [],
        })
        assert resp.status_code == 404

    def test_delete_nonexistent_returns_404(self, client):
        resp = client.delete("/api/groups/nope")
        assert resp.status_code == 404

    def test_create_duplicate_returns_409(self, client):
        client.post("/api/groups", json={"id": "grp1", "name": "G1", "services": []})
        resp = client.post("/api/groups", json={"id": "grp1", "name": "G1 dup", "services": []})
        assert resp.status_code == 409

    def test_persistence(self, client, groups_file):
        client.post("/api/groups", json={"id": "grp1", "name": "G1", "services": []})
        with open(groups_file) as f:
            data = json.load(f)
        assert len(data["groups"]) == 1
```

**Step 2: Run tests — verify they fail**

```bash
CONFIG_PATH=config.json pytest tests/test_server_groups.py -v
```
Expected: FAIL

**Step 3: Implement group store and routes**

`server/group_store.py`:
```python
import json
import os
import threading


class GroupStore:
    _instance = None
    _lock = threading.Lock()

    def __new__(cls):
        if cls._instance is None:
            cls._instance = super().__new__(cls)
            cls._instance._path = os.environ.get("GROUPS_PATH", "/app/groups.json")
            cls._instance._data = cls._instance._load()
        return cls._instance

    def _load(self) -> dict:
        try:
            with open(self._path) as f:
                return json.load(f)
        except (FileNotFoundError, json.JSONDecodeError):
            return {"groups": []}

    def _save(self):
        with open(self._path, "w") as f:
            json.dump(self._data, f, indent=2)

    def list_groups(self) -> list[dict]:
        return self._data["groups"]

    def get_group(self, group_id: str) -> dict | None:
        for g in self._data["groups"]:
            if g["id"] == group_id:
                return g
        return None

    def create_group(self, group: dict) -> dict:
        with self._lock:
            self._data["groups"].append(group)
            self._save()
        return group

    def update_group(self, group_id: str, group: dict) -> dict | None:
        with self._lock:
            for i, g in enumerate(self._data["groups"]):
                if g["id"] == group_id:
                    self._data["groups"][i] = group
                    self._save()
                    return group
        return None

    def delete_group(self, group_id: str) -> bool:
        with self._lock:
            before = len(self._data["groups"])
            self._data["groups"] = [g for g in self._data["groups"] if g["id"] != group_id]
            if len(self._data["groups"]) < before:
                self._save()
                return True
        return False
```

`server/routes/groups.py`:
```python
import httpx
from fastapi import APIRouter, HTTPException

from server.group_store import GroupStore
from server.config import get_node_url
from server.models import Group

router = APIRouter()


def _store():
    return GroupStore()


@router.get("/api/groups")
def list_groups():
    return _store().list_groups()


@router.post("/api/groups")
def create_group(group: Group):
    store = _store()
    if store.get_group(group.id):
        raise HTTPException(status_code=409, detail=f"Group {group.id} already exists")
    return store.create_group(group.model_dump())


@router.put("/api/groups/{group_id}")
def update_group(group_id: str, group: Group):
    result = _store().update_group(group_id, group.model_dump())
    if result is None:
        raise HTTPException(status_code=404, detail=f"Group {group_id} not found")
    return result


@router.delete("/api/groups/{group_id}")
def delete_group(group_id: str):
    if not _store().delete_group(group_id):
        raise HTTPException(status_code=404, detail=f"Group {group_id} not found")
    return {"success": True}


@router.post("/api/groups/{group_id}/start")
async def start_group(group_id: str):
    group = _store().get_group(group_id)
    if not group:
        raise HTTPException(status_code=404, detail=f"Group {group_id} not found")
    results = []
    for svc in group["services"]:
        base = get_node_url(svc["node"])
        if not base:
            results.append({"container": svc["container"], "success": False, "message": f"Node {svc['node']} not found"})
            continue
        try:
            async with httpx.AsyncClient(timeout=30.0) as client:
                resp = await client.post(f"{base}/containers/{svc['container']}/start")
                results.append({"container": svc["container"], **resp.json()})
        except Exception as e:
            results.append({"container": svc["container"], "success": False, "message": str(e)})
    return results


@router.post("/api/groups/{group_id}/stop")
async def stop_group(group_id: str):
    group = _store().get_group(group_id)
    if not group:
        raise HTTPException(status_code=404, detail=f"Group {group_id} not found")
    results = []
    for svc in group["services"]:
        base = get_node_url(svc["node"])
        if not base:
            results.append({"container": svc["container"], "success": False, "message": f"Node {svc['node']} not found"})
            continue
        try:
            async with httpx.AsyncClient(timeout=30.0) as client:
                resp = await client.post(f"{base}/containers/{svc['container']}/stop")
                results.append({"container": svc["container"], **resp.json()})
        except Exception as e:
            results.append({"container": svc["container"], "success": False, "message": str(e)})
    return results


@router.post("/api/groups/{group_id}/restart")
async def restart_group(group_id: str):
    group = _store().get_group(group_id)
    if not group:
        raise HTTPException(status_code=404, detail=f"Group {group_id} not found")
    results = []
    for svc in group["services"]:
        base = get_node_url(svc["node"])
        if not base:
            results.append({"container": svc["container"], "success": False, "message": f"Node {svc['node']} not found"})
            continue
        try:
            async with httpx.AsyncClient(timeout=30.0) as client:
                resp = await client.post(f"{base}/containers/{svc['container']}/restart")
                results.append({"container": svc["container"], **resp.json()})
        except Exception as e:
            results.append({"container": svc["container"], "success": False, "message": str(e)})
    return results
```

Update `server/main.py` — add groups router:
```python
from server.routes.groups import router as groups_router
app.include_router(groups_router)
```

**Step 4: Run tests — verify they pass**

```bash
CONFIG_PATH=config.json pytest tests/test_server_groups.py -v
```
Expected: All PASS

**Step 5: Commit**

```bash
git add server/group_store.py server/routes/groups.py tests/test_server_groups.py server/main.py
git commit -m "feat(server): Group CRUD with JSON persistence and bulk actions (TDD)

Closes #4, #5, progresses #13"
git push origin main
```

---

### Task 7: Dashboard — HTML, CSS, JavaScript

**Gitea issues:** #6, #7, #8, #9, #10

**Files:**
- Create: `server/static/index.html`
- Create: `server/static/style.css`
- Create: `server/static/app.js`

This is the frontend task. Since it's vanilla HTML/CSS/JS with no build step, TDD doesn't apply in the traditional sense. The tests are the API contract tests already written. Build the UI to consume the API endpoints.

**Step 1: Create index.html**

`server/static/index.html` — Single page layout with:
- Sidebar: nodes list, groups list, "All Services" link, "Manage Groups" button
- Main area: service card grid with status badges and action buttons
- Slide-out log panel on the right
- Group editor modal
- Confirmation modal for destructive group actions
- Toast notifications for action feedback

**Step 2: Create style.css**

`server/static/style.css` — Dark theme using CSS variables:
- `--bg-primary: #1a1a2e` (dark blue-black)
- `--bg-secondary: #16213e` (card background)
- `--bg-tertiary: #0f3460` (sidebar)
- `--accent: #00d2ff` (cyan accent)
- `--success: #00c853`, `--danger: #ff1744`, `--warning: #ffab00`
- Status badges: green (running), red (exited), yellow (restarting), gray (paused)
- Responsive grid for service cards
- Slide-out panel animation for logs

**Step 3: Create app.js**

`server/static/app.js` — Vanilla JS application:
- `loadNodes()` — Fetch `/api/nodes`, render sidebar with health dots
- `loadServices(filter)` — Fetch `/api/services`, render card grid, optionally filter by group
- `loadGroups()` — Fetch `/api/groups`, render sidebar group list
- `performAction(node, containerId, action)` — POST to `/api/services/{node}/{id}/{action}`, show toast, refresh
- `loadLogs(node, containerId)` — GET `/api/services/{node}/{id}/logs`, open slide-out panel, auto-scroll
- `groupAction(groupId, action)` — POST to `/api/groups/{id}/{action}` with confirmation modal
- `openGroupEditor(groupId?)` — Modal for create/edit group, checkboxes for services
- `deleteGroup(groupId)` — DELETE with confirmation
- Auto-refresh service status every 15 seconds
- Toast notification system for action feedback

**Step 4: Commit**

```bash
git add server/static/
git commit -m "feat(dashboard): service grid, log viewer, group management, dark theme

Closes #6, #7, #8, #9"
git push origin main
```

---

### Task 8: Dockerfiles and Compose Files

**Gitea issues:** #2, #11

**Files:**
- Create: `server/Dockerfile`
- Create: `deploy/agent/docker-compose.yml`
- Create: `deploy/server/docker-compose.yml`

**Step 1: Create server Dockerfile**

`server/Dockerfile`:
```dockerfile
FROM python:3.12-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8888

CMD ["uvicorn", "server.main:app", "--host", "0.0.0.0", "--port", "8888"]
```

**Step 2: Create agent compose file (template for all nodes)**

`deploy/agent/docker-compose.yml`:
```yaml
services:
  farm-agent:
    image: 127.0.0.1:5050/farm-agent:latest
    container_name: farm-agent
    restart: unless-stopped
    ports:
      - "8889:8889"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    mem_limit: 128M
    memswap_limit: 128M
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:8889/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
```

**Step 3: Create server compose file**

`deploy/server/docker-compose.yml`:
```yaml
services:
  farm-manager:
    image: 127.0.0.1:5050/farm-manager:latest
    container_name: farm-manager
    restart: unless-stopped
    ports:
      - "8888:8888"
    environment:
      - CONFIG_PATH=/app/config.json
      - GROUPS_PATH=/data/groups.json
    volumes:
      - /mnt/docker-data/configs/farm-manager:/data
    mem_limit: 256M
    memswap_limit: 256M
    healthcheck:
      test: ["CMD-SHELL", "curl -sf http://localhost:8888/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 15s
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "3"
```

**Step 4: Commit**

```bash
git add server/Dockerfile deploy/
git commit -m "feat: Dockerfiles and compose files for agent and server

Closes #2, #11"
git push origin main
```

---

### Task 9: Build, Push, and Deploy

**Gitea issues:** #2, #11, #10

**Step 1: Build and push agent image**

On hf-pdocker-01 (has the registry at 127.0.0.1:5050):
```bash
cd /tmp/farm-manager
docker build -t 127.0.0.1:5050/farm-agent:latest -f agent/Dockerfile agent/
docker push 127.0.0.1:5050/farm-agent:latest
```

**Step 2: Build and push server image**

```bash
docker build -t 127.0.0.1:5050/farm-manager:latest -f server/Dockerfile .
docker push 127.0.0.1:5050/farm-manager:latest
```

**Step 3: Deploy agents to all 3 nodes**

Copy compose file to NAS:
```bash
mkdir -p /mnt/docker-data/compose/hf-pdocker-01/farm-agent
mkdir -p /mnt/docker-data/compose/hf-pdocker-02/farm-agent
mkdir -p /mnt/docker-data/compose/bart/farm-agent
cp deploy/agent/docker-compose.yml /mnt/docker-data/compose/hf-pdocker-01/farm-agent/
cp deploy/agent/docker-compose.yml /mnt/docker-data/compose/hf-pdocker-02/farm-agent/
cp deploy/agent/docker-compose.yml /mnt/docker-data/compose/bart/farm-agent/
```

On each node:
```bash
cd /mnt/docker-data/compose/{node}/farm-agent && docker compose up -d
```

**Step 4: Create config directory and copy default groups**

```bash
mkdir -p /mnt/docker-data/configs/farm-manager
cp groups.json /mnt/docker-data/configs/farm-manager/groups.json
```

**Step 5: Deploy server on hf-pdocker-01**

```bash
mkdir -p /mnt/docker-data/compose/hf-pdocker-01/farm-manager
cp deploy/server/docker-compose.yml /mnt/docker-data/compose/hf-pdocker-01/farm-manager/
cd /mnt/docker-data/compose/hf-pdocker-01/farm-manager && docker compose up -d
```

**Step 6: Verify all services are healthy**

```bash
curl -sf http://192.168.86.192:8888/health
curl -sf http://192.168.86.192:8889/health
curl -sf http://192.168.86.100:8889/health
curl -sf http://192.168.86.167:8889/health
```

**Step 7: Commit deploy configs to repo**

```bash
git add deploy/
git commit -m "chore: deploy configs for all nodes

Closes #10, #11"
git push origin main
```

---

### Task 10: Monitoring and Documentation

**Gitea issues:** #15, #16

**Step 1: Add Prometheus blackbox probes**

Add to `stacks/monitoring/prometheus.yml` in the appropriate job sections:
- `http://192.168.86.192:8888/health` (Farm Manager server — http-pdocker01)
- `http://192.168.86.192:8889/health` (Farm Agent — http-pdocker01)
- `http://192.168.86.100:8889/health` (Farm Agent — http-pdocker02)
- `http://192.168.86.167:8889/health` (Farm Agent — http-bart)

Hot-reload Prometheus.

**Step 2: Update CLAUDE.md**

Add Farm Manager to hf-pdocker-01 service list and port reference (8888).
Add Farm Agent to all 3 nodes.

**Step 3: Update MEMORY.md**

Add Farm Manager entry.

**Step 4: Update wiki**

Launch documentation-keeper agent to add Farm Manager page.

**Step 5: Write README.md in repo**

Cover: what it does, architecture diagram, API reference, deployment instructions.

**Step 6: Commit**

```bash
git add README.md
git commit -m "docs: README with architecture and API reference

Closes #15, #16"
git push origin main
```

**Step 7: Close all Gitea issues and milestone**

Use Gitea API to close milestone v1.0.