# Deployment Brief: vLLM serving Hermes 4.3 for OpenClaw/Pluto **Audience:** Claude Code, executing autonomously on Jeremy's homelab. **Goal:** Stand up a vLLM OpenAI-compatible inference server hosting NousResearch Hermes 4.3, with native Hermes tool-call parsing, and wire it into the existing OpenClaw gateway as Pluto's reasoning model. --- ## 0. Why this exists (context you need) We tried serving Hermes 4.3 36B (Q4_K_M GGUF) through Ollama on the Windows box. The model loads and runs at ~98% GPU after forcing `num_gpu 64`, but **structured tool calling fails** — Ollama returns empty `content` and no `tool_calls` array, because Ollama's parser doesn't extract Hermes's `` XML format into the OpenAI-structured field on a self-imported GGUF. This is a known, recurring Ollama sharp edge. vLLM solves this directly: the `--tool-call-parser hermes` flag parses the XML `` format natively into the OpenAI-compatible response schema, no post-processing. That is the entire reason for this migration. **Do not** spend effort trying to fix the Ollama path. The decision to move to vLLM is final. --- ## 1. Hard constraints — read before doing anything 1. **The GPU is in the Windows desktop, NOT the Linux swarm.** - Host: `DESKTOP-5AHPRKR` — ASUS Z790 (TUF GAMING Z790-PLUS WIFI), i7-13700KF, 32GB RAM, **RTX 3090 (24GB VRAM)**. - The Docker Swarm nodes (`hf-pdocker-01`, `hf-pdocker-02`, `bart`) are CPU-only and **cannot** run a 36B model usefully. Do not attempt to schedule vLLM onto the swarm. - vLLM is **Linux/CUDA only**. On Windows this means running inside **WSL2** (Ubuntu) with CUDA passthrough, using the official vLLM Docker image via Docker Desktop's WSL2 backend, OR a native WSL2 Python install. Docker image is preferred for reproducibility. 2. **24GB VRAM is the binding limit.** Hermes 4.3 36B in FP16 is ~72GB — impossible here. Options, in order of preference: - **(A)** A pre-quantized **AWQ or GPTQ 4-bit** build of Hermes 4.3 36B (~18-20GB weights). Tight but workable with `--gpu-memory-utilization 0.95`, `--max-model-len 8192`, and `--enforce-eager` (skip CUDA graph capture to save VRAM). **First choice if a quant exists.** - **(B)** If no 36B AWQ/GPTQ exists, fall back to **Hermes 4.3 in a smaller parameter size** if Nous published one (e.g. a 14B), served in FP16 or AWQ — fits comfortably, full context, fast. **Strongly preferred over fighting the 36B on 24GB.** - **(C)** Last resort: serve the **GGUF Q4** through vLLM (vLLM supports GGUF). Weaker than AWQ in vLLM and partially defeats the point, but the hermes parser still works. Only if A and B are unavailable. 3. **WSL2 will fight you for RAM.** Jeremy's box already had `VmmemWSL` consuming ~13GB. There may now be a `C:\Users\Jeremy\.wslconfig` capping it. Check it. vLLM's model loading stages through system RAM, so ensure WSL2's memory cap is high enough (≥24GB) for the load, OR raise it temporarily. Coordinate: Docker Desktop restarts WSL2 automatically, so don't rely on `wsl --shutdown` to free RAM permanently. 4. **Stop the Ollama server before testing** to avoid port/VRAM contention. Ollama holds VRAM while a model is resident. `ollama stop hermes` and confirm `ollama ps` is empty, or quit the Ollama app entirely. vLLM and Ollama cannot both hold the 3090 at once. --- ## 2. Pre-flight verification (do these first, report results) Run inside WSL2 (Ubuntu): ```bash # Confirm CUDA reaches WSL2 and sees the 3090 nvidia-smi # Expect: "NVIDIA GeForce RTX 3090", 24576 MiB total, driver present. # If this fails, CUDA-on-WSL2 isn't set up — STOP and report. # Confirm Docker can see the GPU docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi # Expect same output from inside a container. ``` If `nvidia-smi` works on the Windows side but not in WSL2, the fix is installing the WSL2 CUDA toolkit (NOT a full driver — the Windows driver provides the WSL2 GPU stub). Report before proceeding so Jeremy can confirm. Also check current RAM headroom and the WSL config: ```bash cat /mnt/c/Users/Jeremy/.wslconfig 2>/dev/null || echo "no .wslconfig" free -h ``` --- ## 3. Model selection (decide, then download) **Step 3a — Find the best-fitting quant.** Search HuggingFace for, in priority order: 1. `Hermes-4.3-36B` AWQ (search orgs: NousResearch, and community quantizers like `casperhansen`, `solidrust`, `bartowski` for AWQ/GPTQ variants) 2. A smaller Hermes 4.3 (14B-class) if one exists — this is genuinely the easier, faster path on 24GB 3. GGUF Q4 as fallback (we already have `hf.co/NousResearch/Hermes-4.3-36B-GGUF:Q4_K_M`) **Report what you find and which option (A/B/C from §1.2) you're proceeding with before downloading 20GB.** If a 36B AWQ exists, confirm its repo name. If the only realistic fit is a smaller Hermes 4.3, say so — Jeremy explicitly prefers a model that runs 100% on-GPU with full context over grinding a 36B into a too-small card. **Step 3b — Download to a persistent WSL2 path** (not the container, so it survives restarts): ```bash mkdir -p ~/models pip install -U "huggingface_hub[cli]" huggingface-cli download --local-dir ~/models/hermes-4.3 ``` --- ## 4. Launch vLLM (Docker, OpenAI-compatible) Create `~/vllm-hermes/docker-compose.yml`. Template for **Option A (36B AWQ)** — adjust `--quantization`, `--model`, and `--max-model-len` per the resolved model: ```yaml services: vllm-hermes: image: vllm/vllm-openai:latest container_name: vllm-hermes runtime: nvidia deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu] ports: - "8000:8000" volumes: - ~/models/hermes-4.3:/models/hermes-4.3 ipc: host command: > --model /models/hermes-4.3 --served-model-name hermes --quantization awq --gpu-memory-utilization 0.95 --max-model-len 8192 --enforce-eager --enable-auto-tool-choice --tool-call-parser hermes --host 0.0.0.0 --port 8000 restart: unless-stopped ``` Key flag rationale: - `--enable-auto-tool-choice --tool-call-parser hermes` — **the whole point.** Native Hermes XML tool-call parsing into OpenAI schema. - `--quantization awq` — set to `gptq` for GPTQ, omit entirely for GGUF (use `--quantization gguf` and point `--model` at the .gguf file) or for an unquantized smaller model. - `--enforce-eager` — disables CUDA graph capture, saving ~1-3GB VRAM. **Critical on the tight 24GB 36B path.** Remove it if you have headroom (smaller model) for a latency improvement. - `--max-model-len 8192` — keep KV cache small to fit. Raise cautiously if VRAM allows; watch for OOM on startup. - `--gpu-memory-utilization 0.95` — aggressive, appropriate for a dedicated inference box. Drop to 0.90 if the desktop also drives a display off the 3090 (it does — there's a monitor at 1680x1050 on it; budget ~0.5-1GB for the desktop, so 0.90 may be safer). Launch and watch logs: ```bash cd ~/vllm-hermes docker compose up -d docker compose logs -f vllm-hermes # Wait for: "Application startup complete" and "Uvicorn running on http://0.0.0.0:8000" ``` **If it OOMs on startup:** lower `--max-model-len` to 4096, confirm `--enforce-eager` is present, drop `--gpu-memory-utilization` to 0.90. If it still won't fit, that's the signal Option A is not viable on this card — fall back to Option B (smaller Hermes 4.3) and report. --- ## 5. Verification — the test that actually matters This is the exact test that FAILED on Ollama. It must pass on vLLM. ```bash curl -s http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "hermes", "messages": [{"role": "user", "content": "What is 8347 multiplied by 291? Use the calculator tool."}], "tools": [{ "type": "function", "function": { "name": "calculator", "description": "Multiply two numbers", "parameters": { "type": "object", "properties": {"a": {"type": "number"}, "b": {"type": "number"}}, "required": ["a", "b"] } } }], "tool_choice": "auto" }' | python3 -m json.tool ``` **PASS criteria:** the response `choices[0].message` has a populated `tool_calls` array containing `function.name == "calculator"` and `function.arguments` parsing to `{"a": 8347, "b": 291}` (or equivalent), with `content` empty/null. That is structured tool calling working. **FAIL:** if `tool_calls` is null/absent and the JSON is dumped in `content` as a string, the parser flag didn't take — verify `--tool-call-parser hermes` is in the running command (`docker compose logs` shows the args at boot) and that the model's chat template actually emits `` tags. Report the raw response. Also sanity-check throughput: ```bash # Plain generation latency curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \ -d '{"model":"hermes","messages":[{"role":"user","content":"Say hello in one sentence."}]}' \ | python3 -m json.tool nvidia-smi # confirm VRAM occupancy and that it's the vLLM process ``` --- ## 6. Wire into OpenClaw (Pluto's brain) OpenClaw is model-agnostic and speaks the OpenAI API. Point it at the vLLM endpoint. The vLLM server is on the Windows box at `http://:8000/v1` (or `http://localhost:8000/v1` if OpenClaw runs on the same Windows box; confirm where Pluto's gateway actually runs — it may be on the swarm). In OpenClaw's model/provider config: - **Base URL:** `http://:8000/v1` - **Model name:** `hermes` (matches `--served-model-name`) - **API key:** any non-empty string (vLLM ignores it by default; use `EMPTY` or `sk-noauth`) **Networking notes:** - vLLM in the WSL2 container binds `0.0.0.0:8000`, but WSL2 networking means the port must be reachable from the Windows host and the LAN. WSL2 usually auto-forwards localhost; for LAN access you may need a `netsh interface portproxy` rule on Windows forwarding the host's LAN IP:8000 to the WSL2 IP:8000, plus a Windows Firewall inbound rule for port 8000 **scoped to the LAN subnet only**. Set this up and verify from another machine with `curl http://:8000/v1/models`. - Do **not** expose port 8000 to the internet. If remote access is wanted later, that goes behind Traefik with forward-auth on the swarm, tunneled to the Windows box — a separate task, not part of this brief. After wiring, restart the OpenClaw gateway and run a Pluto interaction that should trigger a tool (whatever real tools Pluto has registered) to confirm end-to-end tool execution, not just the synthetic calculator test. --- ## 7. Operational notes / gotchas - **Display + inference share the 3090.** The desktop's monitor runs off the same card. Under VRAM pressure this can cause display glitches or vLLM OOM. The `0.90` utilization setting accounts for this; if Jeremy ever runs the box headless (RDP only, which the earlier session showed — there was a Remote Display Adapter), more VRAM frees up and utilization can go to 0.95+. - **vLLM holds VRAM for its lifetime.** It does not unload like Ollama. That's expected and desirable for an always-on Pluto brain, but it means the 3090 is committed — no gaming / other GPU work while it runs. - **Auto-start on boot:** `restart: unless-stopped` handles container restarts, but WSL2 + Docker Desktop must themselves start on Windows login. For true always-on, enable Docker Desktop "Start on login" and consider WSL2 boot via Task Scheduler, OR (better long-term) move this whole stack to a dedicated headless Linux box with the 3090 if one becomes available. Flag this to Jeremy as a follow-up if always-on reliability matters for VoiceCoach. - **Context length is capped at 8192 here** for VRAM reasons. Hermes 4.3 supports far more (trained to 512K), but you can't afford the KV cache on 24GB. If Pluto workflows need longer context, that's a hardware conversation (bigger card / second GPU), not a config tweak. - **Version pinning:** `vllm/vllm-openai:latest` is convenient but pin to a specific tag for production stability once verified working, so an image update doesn't silently break the tool parser. --- ## 8. Report back to Jeremy with 1. `nvidia-smi` output from inside WSL2 (GPU visible y/n). 2. Which model option (A/B/C) you proceeded with and the exact repo ID. 3. The full JSON from the §5 calculator tool-call test (the pass/fail evidence). 4. Measured tokens/sec and VRAM occupancy from `nvidia-smi`. 5. Whether OpenClaw end-to-end tool execution worked with the real Pluto toolset. 6. Any networking steps needed for LAN access (portproxy/firewall rules added). Do not mark this complete until the §5 tool-call test passes with a structured `tool_calls` array. That single result is the definition of done for the inference layer.