hobbs-iac/vllm-hermes-deployment-brief.md

# Deployment Brief: vLLM serving Hermes 4.3 for OpenClaw/Pluto

**Audience:** Claude Code, executing autonomously on Jeremy's homelab.
**Goal:** Stand up a vLLM OpenAI-compatible inference server hosting NousResearch Hermes 4.3, with native Hermes tool-call parsing, and wire it into the existing OpenClaw gateway as Pluto's reasoning model.

---

## 0. Why this exists (context you need)

We tried serving Hermes 4.3 36B (Q4_K_M GGUF) through Ollama on the Windows box. The model loads and runs at ~98% GPU after forcing `num_gpu 64`, but **structured tool calling fails** — Ollama returns empty `content` and no `tool_calls` array, because Ollama's parser doesn't extract Hermes's `<tool_call>` XML format into the OpenAI-structured field on a self-imported GGUF. This is a known, recurring Ollama sharp edge.

vLLM solves this directly: the `--tool-call-parser hermes` flag parses the XML `<tool_call>` format natively into the OpenAI-compatible response schema, no post-processing. That is the entire reason for this migration.

**Do not** spend effort trying to fix the Ollama path. The decision to move to vLLM is final.

---

## 1. Hard constraints — read before doing anything

1. **The GPU is in the Windows desktop, NOT the Linux swarm.**
   - Host: `DESKTOP-5AHPRKR` — ASUS Z790 (TUF GAMING Z790-PLUS WIFI), i7-13700KF, 32GB RAM, **RTX 3090 (24GB VRAM)**.
   - The Docker Swarm nodes (`hf-pdocker-01`, `hf-pdocker-02`, `bart`) are CPU-only and **cannot** run a 36B model usefully. Do not attempt to schedule vLLM onto the swarm.
   - vLLM is **Linux/CUDA only**. On Windows this means running inside **WSL2** (Ubuntu) with CUDA passthrough, using the official vLLM Docker image via Docker Desktop's WSL2 backend, OR a native WSL2 Python install. Docker image is preferred for reproducibility.

2. **24GB VRAM is the binding limit.** Hermes 4.3 36B in FP16 is ~72GB — impossible here. Options, in order of preference:
   - **(A)** A pre-quantized **AWQ or GPTQ 4-bit** build of Hermes 4.3 36B (~18-20GB weights). Tight but workable with `--gpu-memory-utilization 0.95`, `--max-model-len 8192`, and `--enforce-eager` (skip CUDA graph capture to save VRAM). **First choice if a quant exists.**
   - **(B)** If no 36B AWQ/GPTQ exists, fall back to **Hermes 4.3 in a smaller parameter size** if Nous published one (e.g. a 14B), served in FP16 or AWQ — fits comfortably, full context, fast. **Strongly preferred over fighting the 36B on 24GB.**
   - **(C)** Last resort: serve the **GGUF Q4** through vLLM (vLLM supports GGUF). Weaker than AWQ in vLLM and partially defeats the point, but the hermes parser still works. Only if A and B are unavailable.

3. **WSL2 will fight you for RAM.** Jeremy's box already had `VmmemWSL` consuming ~13GB. There may now be a `C:\Users\Jeremy\.wslconfig` capping it. Check it. vLLM's model loading stages through system RAM, so ensure WSL2's memory cap is high enough (≥24GB) for the load, OR raise it temporarily. Coordinate: Docker Desktop restarts WSL2 automatically, so don't rely on `wsl --shutdown` to free RAM permanently.

4. **Stop the Ollama server before testing** to avoid port/VRAM contention. Ollama holds VRAM while a model is resident. `ollama stop hermes` and confirm `ollama ps` is empty, or quit the Ollama app entirely. vLLM and Ollama cannot both hold the 3090 at once.

---

## 2. Pre-flight verification (do these first, report results)

Run inside WSL2 (Ubuntu):

```bash
# Confirm CUDA reaches WSL2 and sees the 3090
nvidia-smi
# Expect: "NVIDIA GeForce RTX 3090", 24576 MiB total, driver present.
# If this fails, CUDA-on-WSL2 isn't set up — STOP and report.

# Confirm Docker can see the GPU
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
# Expect same output from inside a container.
```

If `nvidia-smi` works on the Windows side but not in WSL2, the fix is installing the WSL2 CUDA toolkit (NOT a full driver — the Windows driver provides the WSL2 GPU stub). Report before proceeding so Jeremy can confirm.

Also check current RAM headroom and the WSL config:

```bash
cat /mnt/c/Users/Jeremy/.wslconfig 2>/dev/null || echo "no .wslconfig"
free -h
```

---

## 3. Model selection (decide, then download)

**Step 3a — Find the best-fitting quant.** Search HuggingFace for, in priority order:
1. `Hermes-4.3-36B` AWQ (search orgs: NousResearch, and community quantizers like `casperhansen`, `solidrust`, `bartowski` for AWQ/GPTQ variants)
2. A smaller Hermes 4.3 (14B-class) if one exists — this is genuinely the easier, faster path on 24GB
3. GGUF Q4 as fallback (we already have `hf.co/NousResearch/Hermes-4.3-36B-GGUF:Q4_K_M`)

**Report what you find and which option (A/B/C from §1.2) you're proceeding with before downloading 20GB.** If a 36B AWQ exists, confirm its repo name. If the only realistic fit is a smaller Hermes 4.3, say so — Jeremy explicitly prefers a model that runs 100% on-GPU with full context over grinding a 36B into a too-small card.

**Step 3b — Download to a persistent WSL2 path** (not the container, so it survives restarts):

```bash
mkdir -p ~/models
pip install -U "huggingface_hub[cli]"
huggingface-cli download <RESOLVED_REPO_ID> --local-dir ~/models/hermes-4.3
```

---

## 4. Launch vLLM (Docker, OpenAI-compatible)

Create `~/vllm-hermes/docker-compose.yml`. Template for **Option A (36B AWQ)** — adjust `--quantization`, `--model`, and `--max-model-len` per the resolved model:

```yaml
services:
  vllm-hermes:
    image: vllm/vllm-openai:latest
    container_name: vllm-hermes
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    volumes:
      - ~/models/hermes-4.3:/models/hermes-4.3
    ipc: host
    command: >
      --model /models/hermes-4.3
      --served-model-name hermes
      --quantization awq
      --gpu-memory-utilization 0.95
      --max-model-len 8192
      --enforce-eager
      --enable-auto-tool-choice
      --tool-call-parser hermes
      --host 0.0.0.0
      --port 8000
    restart: unless-stopped
```

Key flag rationale:
- `--enable-auto-tool-choice --tool-call-parser hermes` — **the whole point.** Native Hermes XML tool-call parsing into OpenAI schema.
- `--quantization awq` — set to `gptq` for GPTQ, omit entirely for GGUF (use `--quantization gguf` and point `--model` at the .gguf file) or for an unquantized smaller model.
- `--enforce-eager` — disables CUDA graph capture, saving ~1-3GB VRAM. **Critical on the tight 24GB 36B path.** Remove it if you have headroom (smaller model) for a latency improvement.
- `--max-model-len 8192` — keep KV cache small to fit. Raise cautiously if VRAM allows; watch for OOM on startup.
- `--gpu-memory-utilization 0.95` — aggressive, appropriate for a dedicated inference box. Drop to 0.90 if the desktop also drives a display off the 3090 (it does — there's a monitor at 1680x1050 on it; budget ~0.5-1GB for the desktop, so 0.90 may be safer).

Launch and watch logs:

```bash
cd ~/vllm-hermes
docker compose up -d
docker compose logs -f vllm-hermes
# Wait for: "Application startup complete" and "Uvicorn running on http://0.0.0.0:8000"
```

**If it OOMs on startup:** lower `--max-model-len` to 4096, confirm `--enforce-eager` is present, drop `--gpu-memory-utilization` to 0.90. If it still won't fit, that's the signal Option A is not viable on this card — fall back to Option B (smaller Hermes 4.3) and report.

---

## 5. Verification — the test that actually matters

This is the exact test that FAILED on Ollama. It must pass on vLLM.

```bash
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hermes",
    "messages": [{"role": "user", "content": "What is 8347 multiplied by 291? Use the calculator tool."}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "calculator",
        "description": "Multiply two numbers",
        "parameters": {
          "type": "object",
          "properties": {"a": {"type": "number"}, "b": {"type": "number"}},
          "required": ["a", "b"]
        }
      }
    }],
    "tool_choice": "auto"
  }' | python3 -m json.tool
```

**PASS criteria:** the response `choices[0].message` has a populated `tool_calls` array containing `function.name == "calculator"` and `function.arguments` parsing to `{"a": 8347, "b": 291}` (or equivalent), with `content` empty/null. That is structured tool calling working.

**FAIL:** if `tool_calls` is null/absent and the JSON is dumped in `content` as a string, the parser flag didn't take — verify `--tool-call-parser hermes` is in the running command (`docker compose logs` shows the args at boot) and that the model's chat template actually emits `<tool_call>` tags. Report the raw response.

Also sanity-check throughput:

```bash
# Plain generation latency
curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"hermes","messages":[{"role":"user","content":"Say hello in one sentence."}]}' \
  | python3 -m json.tool
nvidia-smi  # confirm VRAM occupancy and that it's the vLLM process
```

---

## 6. Wire into OpenClaw (Pluto's brain)

OpenClaw is model-agnostic and speaks the OpenAI API. Point it at the vLLM endpoint. The vLLM server is on the Windows box at `http://<WINDOWS_LAN_IP>:8000/v1` (or `http://localhost:8000/v1` if OpenClaw runs on the same Windows box; confirm where Pluto's gateway actually runs — it may be on the swarm).

In OpenClaw's model/provider config:
- **Base URL:** `http://<WINDOWS_LAN_IP>:8000/v1`
- **Model name:** `hermes` (matches `--served-model-name`)
- **API key:** any non-empty string (vLLM ignores it by default; use `EMPTY` or `sk-noauth`)

**Networking notes:**
- vLLM in the WSL2 container binds `0.0.0.0:8000`, but WSL2 networking means the port must be reachable from the Windows host and the LAN. WSL2 usually auto-forwards localhost; for LAN access you may need a `netsh interface portproxy` rule on Windows forwarding the host's LAN IP:8000 to the WSL2 IP:8000, plus a Windows Firewall inbound rule for port 8000 **scoped to the LAN subnet only**. Set this up and verify from another machine with `curl http://<WINDOWS_LAN_IP>:8000/v1/models`.
- Do **not** expose port 8000 to the internet. If remote access is wanted later, that goes behind Traefik with forward-auth on the swarm, tunneled to the Windows box — a separate task, not part of this brief.

After wiring, restart the OpenClaw gateway and run a Pluto interaction that should trigger a tool (whatever real tools Pluto has registered) to confirm end-to-end tool execution, not just the synthetic calculator test.

---

## 7. Operational notes / gotchas

- **Display + inference share the 3090.** The desktop's monitor runs off the same card. Under VRAM pressure this can cause display glitches or vLLM OOM. The `0.90` utilization setting accounts for this; if Jeremy ever runs the box headless (RDP only, which the earlier session showed — there was a Remote Display Adapter), more VRAM frees up and utilization can go to 0.95+.
- **vLLM holds VRAM for its lifetime.** It does not unload like Ollama. That's expected and desirable for an always-on Pluto brain, but it means the 3090 is committed — no gaming / other GPU work while it runs.
- **Auto-start on boot:** `restart: unless-stopped` handles container restarts, but WSL2 + Docker Desktop must themselves start on Windows login. For true always-on, enable Docker Desktop "Start on login" and consider WSL2 boot via Task Scheduler, OR (better long-term) move this whole stack to a dedicated headless Linux box with the 3090 if one becomes available. Flag this to Jeremy as a follow-up if always-on reliability matters for VoiceCoach.
- **Context length is capped at 8192 here** for VRAM reasons. Hermes 4.3 supports far more (trained to 512K), but you can't afford the KV cache on 24GB. If Pluto workflows need longer context, that's a hardware conversation (bigger card / second GPU), not a config tweak.
- **Version pinning:** `vllm/vllm-openai:latest` is convenient but pin to a specific tag for production stability once verified working, so an image update doesn't silently break the tool parser.

---

## 8. Report back to Jeremy with

1. `nvidia-smi` output from inside WSL2 (GPU visible y/n).
2. Which model option (A/B/C) you proceeded with and the exact repo ID.
3. The full JSON from the §5 calculator tool-call test (the pass/fail evidence).
4. Measured tokens/sec and VRAM occupancy from `nvidia-smi`.
5. Whether OpenClaw end-to-end tool execution worked with the real Pluto toolset.
6. Any networking steps needed for LAN access (portproxy/firewall rules added).

Do not mark this complete until the §5 tool-call test passes with a structured `tool_calls` array. That single result is the definition of done for the inference layer.