hobbs-iac/vllm-hermes-deployment-brief.md

Deployment Brief: vLLM serving Hermes 4.3 for OpenClaw/Pluto

Audience: Claude Code, executing autonomously on Jeremy's homelab. Goal: Stand up a vLLM OpenAI-compatible inference server hosting NousResearch Hermes 4.3, with native Hermes tool-call parsing, and wire it into the existing OpenClaw gateway as Pluto's reasoning model.

---

0. Why this exists (context you need)

We tried serving Hermes 4.3 36B (Q4_K_M GGUF) through Ollama on the Windows box. The model loads and runs at ~98% GPU after forcing num_gpu 64, but structured tool calling fails — Ollama returns empty content and no tool_calls array, because Ollama's parser doesn't extract Hermes's <tool_call> XML format into the OpenAI-structured field on a self-imported GGUF. This is a known, recurring Ollama sharp edge.

vLLM solves this directly: the --tool-call-parser hermes flag parses the XML <tool_call> format natively into the OpenAI-compatible response schema, no post-processing. That is the entire reason for this migration.

Do not spend effort trying to fix the Ollama path. The decision to move to vLLM is final.

---

1. Hard constraints — read before doing anything

1. The GPU is in the Windows desktop, NOT the Linux swarm.

   • Host: DESKTOP-5AHPRKR — ASUS Z790 (TUF GAMING Z790-PLUS WIFI), i7-13700KF, 32GB RAM, RTX 3090 (24GB VRAM).
   • The Docker Swarm nodes (hf-pdocker-01, hf-pdocker-02, bart) are CPU-only and cannot run a 36B model usefully. Do not attempt to schedule vLLM onto the swarm.
   • vLLM is Linux/CUDA only. On Windows this means running inside WSL2 (Ubuntu) with CUDA passthrough, using the official vLLM Docker image via Docker Desktop's WSL2 backend, OR a native WSL2 Python install. Docker image is preferred for reproducibility.

2. 24GB VRAM is the binding limit. Hermes 4.3 36B in FP16 is ~72GB — impossible here. Options, in order of preference:

   • (A) A pre-quantized AWQ or GPTQ 4-bit build of Hermes 4.3 36B (~18-20GB weights). Tight but workable with --gpu-memory-utilization 0.95, --max-model-len 8192, and --enforce-eager (skip CUDA graph capture to save VRAM). First choice if a quant exists.
   • (B) If no 36B AWQ/GPTQ exists, fall back to Hermes 4.3 in a smaller parameter size if Nous published one (e.g. a 14B), served in FP16 or AWQ — fits comfortably, full context, fast. Strongly preferred over fighting the 36B on 24GB.
   • (C) Last resort: serve the GGUF Q4 through vLLM (vLLM supports GGUF). Weaker than AWQ in vLLM and partially defeats the point, but the hermes parser still works. Only if A and B are unavailable.

3. WSL2 will fight you for RAM. Jeremy's box already had VmmemWSL consuming ~13GB. There may now be a C:\Users\Jeremy\.wslconfig capping it. Check it. vLLM's model loading stages through system RAM, so ensure WSL2's memory cap is high enough (≥24GB) for the load, OR raise it temporarily. Coordinate: Docker Desktop restarts WSL2 automatically, so don't rely on wsl --shutdown to free RAM permanently.

4. Stop the Ollama server before testing to avoid port/VRAM contention. Ollama holds VRAM while a model is resident. ollama stop hermes and confirm ollama ps is empty, or quit the Ollama app entirely. vLLM and Ollama cannot both hold the 3090 at once.

---

2. Pre-flight verification (do these first, report results)

Run inside WSL2 (Ubuntu):

# Confirm CUDA reaches WSL2 and sees the 3090
nvidia-smi
# Expect: "NVIDIA GeForce RTX 3090", 24576 MiB total, driver present.
# If this fails, CUDA-on-WSL2 isn't set up — STOP and report.

# Confirm Docker can see the GPU
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
# Expect same output from inside a container.

If nvidia-smi works on the Windows side but not in WSL2, the fix is installing the WSL2 CUDA toolkit (NOT a full driver — the Windows driver provides the WSL2 GPU stub). Report before proceeding so Jeremy can confirm.

Also check current RAM headroom and the WSL config:

cat /mnt/c/Users/Jeremy/.wslconfig 2>/dev/null || echo "no .wslconfig"
free -h

---

3. Model selection (decide, then download)

Step 3a — Find the best-fitting quant. Search HuggingFace for, in priority order:

1. Hermes-4.3-36B AWQ (search orgs: NousResearch, and community quantizers like casperhansen, solidrust, bartowski for AWQ/GPTQ variants)
2. A smaller Hermes 4.3 (14B-class) if one exists — this is genuinely the easier, faster path on 24GB
3. GGUF Q4 as fallback (we already have hf.co/NousResearch/Hermes-4.3-36B-GGUF:Q4_K_M)

Report what you find and which option (A/B/C from §1.2) you're proceeding with before downloading 20GB. If a 36B AWQ exists, confirm its repo name. If the only realistic fit is a smaller Hermes 4.3, say so — Jeremy explicitly prefers a model that runs 100% on-GPU with full context over grinding a 36B into a too-small card.

Step 3b — Download to a persistent WSL2 path (not the container, so it survives restarts):

mkdir -p ~/models
pip install -U "huggingface_hub[cli]"
huggingface-cli download <RESOLVED_REPO_ID> --local-dir ~/models/hermes-4.3

---

4. Launch vLLM (Docker, OpenAI-compatible)

Create ~/vllm-hermes/docker-compose.yml. Template for Option A (36B AWQ) — adjust --quantization, --model, and --max-model-len per the resolved model:

services:
  vllm-hermes:
    image: vllm/vllm-openai:latest
    container_name: vllm-hermes
    runtime: nvidia
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    ports:
      - "8000:8000"
    volumes:
      - ~/models/hermes-4.3:/models/hermes-4.3
    ipc: host
    command: >
      --model /models/hermes-4.3
      --served-model-name hermes
      --quantization awq
      --gpu-memory-utilization 0.95
      --max-model-len 8192
      --enforce-eager
      --enable-auto-tool-choice
      --tool-call-parser hermes
      --host 0.0.0.0
      --port 8000
    restart: unless-stopped

Key flag rationale:

• --enable-auto-tool-choice --tool-call-parser hermes — the whole point. Native Hermes XML tool-call parsing into OpenAI schema.
• --quantization awq — set to gptq for GPTQ, omit entirely for GGUF (use --quantization gguf and point --model at the .gguf file) or for an unquantized smaller model.
• --enforce-eager — disables CUDA graph capture, saving ~1-3GB VRAM. Critical on the tight 24GB 36B path. Remove it if you have headroom (smaller model) for a latency improvement.
• --max-model-len 8192 — keep KV cache small to fit. Raise cautiously if VRAM allows; watch for OOM on startup.
• --gpu-memory-utilization 0.95 — aggressive, appropriate for a dedicated inference box. Drop to 0.90 if the desktop also drives a display off the 3090 (it does — there's a monitor at 1680x1050 on it; budget ~0.5-1GB for the desktop, so 0.90 may be safer).

Launch and watch logs:

cd ~/vllm-hermes
docker compose up -d
docker compose logs -f vllm-hermes
# Wait for: "Application startup complete" and "Uvicorn running on http://0.0.0.0:8000"

If it OOMs on startup: lower --max-model-len to 4096, confirm --enforce-eager is present, drop --gpu-memory-utilization to 0.90. If it still won't fit, that's the signal Option A is not viable on this card — fall back to Option B (smaller Hermes 4.3) and report.

---

5. Verification — the test that actually matters

This is the exact test that FAILED on Ollama. It must pass on vLLM.

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hermes",
    "messages": [{"role": "user", "content": "What is 8347 multiplied by 291? Use the calculator tool."}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "calculator",
        "description": "Multiply two numbers",
        "parameters": {
          "type": "object",
          "properties": {"a": {"type": "number"}, "b": {"type": "number"}},
          "required": ["a", "b"]
        }
      }
    }],
    "tool_choice": "auto"
  }' | python3 -m json.tool

PASS criteria: the response choices[0].message has a populated tool_calls array containing function.name == "calculator" and function.arguments parsing to {"a": 8347, "b": 291} (or equivalent), with content empty/null. That is structured tool calling working.

FAIL: if tool_calls is null/absent and the JSON is dumped in content as a string, the parser flag didn't take — verify --tool-call-parser hermes is in the running command (docker compose logs shows the args at boot) and that the model's chat template actually emits <tool_call> tags. Report the raw response.

Also sanity-check throughput:

# Plain generation latency
curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"hermes","messages":[{"role":"user","content":"Say hello in one sentence."}]}' \
  | python3 -m json.tool
nvidia-smi  # confirm VRAM occupancy and that it's the vLLM process

---

6. Wire into OpenClaw (Pluto's brain)

OpenClaw is model-agnostic and speaks the OpenAI API. Point it at the vLLM endpoint. The vLLM server is on the Windows box at http://<WINDOWS_LAN_IP>:8000/v1 (or http://localhost:8000/v1 if OpenClaw runs on the same Windows box; confirm where Pluto's gateway actually runs — it may be on the swarm).

In OpenClaw's model/provider config:

• Base URL: http://<WINDOWS_LAN_IP>:8000/v1
• Model name: hermes (matches --served-model-name)
• API key: any non-empty string (vLLM ignores it by default; use EMPTY or sk-noauth)

Networking notes:

• vLLM in the WSL2 container binds 0.0.0.0:8000, but WSL2 networking means the port must be reachable from the Windows host and the LAN. WSL2 usually auto-forwards localhost; for LAN access you may need a netsh interface portproxy rule on Windows forwarding the host's LAN IP:8000 to the WSL2 IP:8000, plus a Windows Firewall inbound rule for port 8000 scoped to the LAN subnet only. Set this up and verify from another machine with curl http://<WINDOWS_LAN_IP>:8000/v1/models.
• Do not expose port 8000 to the internet. If remote access is wanted later, that goes behind Traefik with forward-auth on the swarm, tunneled to the Windows box — a separate task, not part of this brief.

After wiring, restart the OpenClaw gateway and run a Pluto interaction that should trigger a tool (whatever real tools Pluto has registered) to confirm end-to-end tool execution, not just the synthetic calculator test.

---

7. Operational notes / gotchas

• Display + inference share the 3090. The desktop's monitor runs off the same card. Under VRAM pressure this can cause display glitches or vLLM OOM. The 0.90 utilization setting accounts for this; if Jeremy ever runs the box headless (RDP only, which the earlier session showed — there was a Remote Display Adapter), more VRAM frees up and utilization can go to 0.95+.
• vLLM holds VRAM for its lifetime. It does not unload like Ollama. That's expected and desirable for an always-on Pluto brain, but it means the 3090 is committed — no gaming / other GPU work while it runs.
• Auto-start on boot: restart: unless-stopped handles container restarts, but WSL2 + Docker Desktop must themselves start on Windows login. For true always-on, enable Docker Desktop "Start on login" and consider WSL2 boot via Task Scheduler, OR (better long-term) move this whole stack to a dedicated headless Linux box with the 3090 if one becomes available. Flag this to Jeremy as a follow-up if always-on reliability matters for VoiceCoach.
• Context length is capped at 8192 here for VRAM reasons. Hermes 4.3 supports far more (trained to 512K), but you can't afford the KV cache on 24GB. If Pluto workflows need longer context, that's a hardware conversation (bigger card / second GPU), not a config tweak.
• Version pinning: vllm/vllm-openai:latest is convenient but pin to a specific tag for production stability once verified working, so an image update doesn't silently break the tool parser.

---

8. Report back to Jeremy with

1. nvidia-smi output from inside WSL2 (GPU visible y/n).
2. Which model option (A/B/C) you proceeded with and the exact repo ID.
3. The full JSON from the §5 calculator tool-call test (the pass/fail evidence).
4. Measured tokens/sec and VRAM occupancy from nvidia-smi.
5. Whether OpenClaw end-to-end tool execution worked with the real Pluto toolset.
6. Any networking steps needed for LAN access (portproxy/firewall rules added).

Do not mark this complete until the §5 tool-call test passes with a structured tool_calls array. That single result is the definition of done for the inference layer.