jeremy/hobbs-iac
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: sync accumulated infra changes and docs

Browse Source 
Removals: openwebui and netclaw compose + Traefik routes (both decommissioned).
Updates: litellm, n8n, vaultwarden, ai-agents, media, traefik routes, CLAUDE.md
(nodes/ports/known-issues), agent definitions.
Additions: Rongorongo tree, LinkedStorm plan, rename-bart spec, vllm brief,
tls-options and security-headers Traefik config.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
jeremy
2026-07-11 11:19:21 -05:00
parent 47ac49a039
commit 27f6d95bf6
34 changed files with 8589 additions and 322 deletions
Download Patch File Download Diff File Expand all files Collapse all files
vllm-hermes-deployment-brief.md
+215
View File
@@ -0,0 +1,215 @@
# Deployment Brief: vLLM serving Hermes 4.3 for OpenClaw/Pluto
**Audience:** Claude Code, executing autonomously on Jeremy's homelab.
**Goal:** Stand up a vLLM OpenAI-compatible inference server hosting NousResearch Hermes 4.3, with native Hermes tool-call parsing, and wire it into the existing OpenClaw gateway as Pluto's reasoning model.
---
## 0. Why this exists (context you need)
We tried serving Hermes 4.3 36B (Q4_K_M GGUF) through Ollama on the Windows box. The model loads and runs at ~98% GPU after forcing `num_gpu 64`, but **structured tool calling fails** — Ollama returns empty `content` and no `tool_calls` array, because Ollama's parser doesn't extract Hermes's `<tool_call>` XML format into the OpenAI-structured field on a self-imported GGUF. This is a known, recurring Ollama sharp edge.
vLLM solves this directly: the `--tool-call-parser hermes` flag parses the XML `<tool_call>` format natively into the OpenAI-compatible response schema, no post-processing. That is the entire reason for this migration.
**Do not** spend effort trying to fix the Ollama path. The decision to move to vLLM is final.
---
## 1. Hard constraints — read before doing anything
1. **The GPU is in the Windows desktop, NOT the Linux swarm.**
- Host: `DESKTOP-5AHPRKR` — ASUS Z790 (TUF GAMING Z790-PLUS WIFI), i7-13700KF, 32GB RAM, **RTX 3090 (24GB VRAM)**.
- The Docker Swarm nodes (`hf-pdocker-01`, `hf-pdocker-02`, `bart`) are CPU-only and **cannot** run a 36B model usefully. Do not attempt to schedule vLLM onto the swarm.
- vLLM is **Linux/CUDA only**. On Windows this means running inside **WSL2** (Ubuntu) with CUDA passthrough, using the official vLLM Docker image via Docker Desktop's WSL2 backend, OR a native WSL2 Python install. Docker image is preferred for reproducibility.
2. **24GB VRAM is the binding limit.** Hermes 4.3 36B in FP16 is ~72GB — impossible here. Options, in order of preference:
- **(A)** A pre-quantized **AWQ or GPTQ 4-bit** build of Hermes 4.3 36B (~18-20GB weights). Tight but workable with `--gpu-memory-utilization 0.95`, `--max-model-len 8192`, and `--enforce-eager` (skip CUDA graph capture to save VRAM). **First choice if a quant exists.**
- **(B)** If no 36B AWQ/GPTQ exists, fall back to **Hermes 4.3 in a smaller parameter size** if Nous published one (e.g. a 14B), served in FP16 or AWQ — fits comfortably, full context, fast. **Strongly preferred over fighting the 36B on 24GB.**
- **(C)** Last resort: serve the **GGUF Q4** through vLLM (vLLM supports GGUF). Weaker than AWQ in vLLM and partially defeats the point, but the hermes parser still works. Only if A and B are unavailable.
3. **WSL2 will fight you for RAM.** Jeremy's box already had `VmmemWSL` consuming ~13GB. There may now be a `C:\Users\Jeremy\.wslconfig` capping it. Check it. vLLM's model loading stages through system RAM, so ensure WSL2's memory cap is high enough (≥24GB) for the load, OR raise it temporarily. Coordinate: Docker Desktop restarts WSL2 automatically, so don't rely on `wsl --shutdown` to free RAM permanently.
4. **Stop the Ollama server before testing** to avoid port/VRAM contention. Ollama holds VRAM while a model is resident. `ollama stop hermes` and confirm `ollama ps` is empty, or quit the Ollama app entirely. vLLM and Ollama cannot both hold the 3090 at once.
---
## 2. Pre-flight verification (do these first, report results)
Run inside WSL2 (Ubuntu):
```bash
# Confirm CUDA reaches WSL2 and sees the 3090
nvidia-smi
# Expect: "NVIDIA GeForce RTX 3090", 24576 MiB total, driver present.
# If this fails, CUDA-on-WSL2 isn't set up — STOP and report.
# Confirm Docker can see the GPU
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
# Expect same output from inside a container.
```
If `nvidia-smi` works on the Windows side but not in WSL2, the fix is installing the WSL2 CUDA toolkit (NOT a full driver — the Windows driver provides the WSL2 GPU stub). Report before proceeding so Jeremy can confirm.
Also check current RAM headroom and the WSL config:
```bash
cat /mnt/c/Users/Jeremy/.wslconfig 2>/dev/null || echo "no .wslconfig"
free -h
```
---
## 3. Model selection (decide, then download)
**Step 3a — Find the best-fitting quant.** Search HuggingFace for, in priority order:
1. `Hermes-4.3-36B` AWQ (search orgs: NousResearch, and community quantizers like `casperhansen`, `solidrust`, `bartowski` for AWQ/GPTQ variants)
2. A smaller Hermes 4.3 (14B-class) if one exists — this is genuinely the easier, faster path on 24GB
3. GGUF Q4 as fallback (we already have `hf.co/NousResearch/Hermes-4.3-36B-GGUF:Q4_K_M`)
**Report what you find and which option (A/B/C from §1.2) you're proceeding with before downloading 20GB.** If a 36B AWQ exists, confirm its repo name. If the only realistic fit is a smaller Hermes 4.3, say so — Jeremy explicitly prefers a model that runs 100% on-GPU with full context over grinding a 36B into a too-small card.
**Step 3b — Download to a persistent WSL2 path** (not the container, so it survives restarts):
```bash
mkdir -p ~/models
pip install -U "huggingface_hub[cli]"
huggingface-cli download <RESOLVED_REPO_ID> --local-dir ~/models/hermes-4.3
```
---
## 4. Launch vLLM (Docker, OpenAI-compatible)
Create `~/vllm-hermes/docker-compose.yml`. Template for **Option A (36B AWQ)** — adjust `--quantization`, `--model`, and `--max-model-len` per the resolved model:
```yaml
services:
vllm-hermes:
image: vllm/vllm-openai:latest
container_name: vllm-hermes
runtime: nvidia
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
ports:
- "8000:8000"
volumes:
- ~/models/hermes-4.3:/models/hermes-4.3
ipc: host
command: >
--model /models/hermes-4.3
--served-model-name hermes
--quantization awq
--gpu-memory-utilization 0.95
--max-model-len 8192
--enforce-eager
--enable-auto-tool-choice
--tool-call-parser hermes
--host 0.0.0.0
--port 8000
restart: unless-stopped
```
Key flag rationale:
- `--enable-auto-tool-choice --tool-call-parser hermes` — **the whole point.** Native Hermes XML tool-call parsing into OpenAI schema.
- `--quantization awq` — set to `gptq` for GPTQ, omit entirely for GGUF (use `--quantization gguf` and point `--model` at the .gguf file) or for an unquantized smaller model.
- `--enforce-eager` — disables CUDA graph capture, saving ~1-3GB VRAM. **Critical on the tight 24GB 36B path.** Remove it if you have headroom (smaller model) for a latency improvement.
- `--max-model-len 8192` — keep KV cache small to fit. Raise cautiously if VRAM allows; watch for OOM on startup.
- `--gpu-memory-utilization 0.95` — aggressive, appropriate for a dedicated inference box. Drop to 0.90 if the desktop also drives a display off the 3090 (it does — there's a monitor at 1680x1050 on it; budget ~0.5-1GB for the desktop, so 0.90 may be safer).
Launch and watch logs:
```bash
cd ~/vllm-hermes
docker compose up -d
docker compose logs -f vllm-hermes
# Wait for: "Application startup complete" and "Uvicorn running on http://0.0.0.0:8000"
```
**If it OOMs on startup:** lower `--max-model-len` to 4096, confirm `--enforce-eager` is present, drop `--gpu-memory-utilization` to 0.90. If it still won't fit, that's the signal Option A is not viable on this card — fall back to Option B (smaller Hermes 4.3) and report.
---
## 5. Verification — the test that actually matters
This is the exact test that FAILED on Ollama. It must pass on vLLM.
```bash
curl -s http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "hermes",
"messages": [{"role": "user", "content": "What is 8347 multiplied by 291? Use the calculator tool."}],
"tools": [{
"type": "function",
"function": {
"name": "calculator",
"description": "Multiply two numbers",
"parameters": {
"type": "object",
"properties": {"a": {"type": "number"}, "b": {"type": "number"}},
"required": ["a", "b"]
}
}
}],
"tool_choice": "auto"
}' | python3 -m json.tool
```
**PASS criteria:** the response `choices[0].message` has a populated `tool_calls` array containing `function.name == "calculator"` and `function.arguments` parsing to `{"a": 8347, "b": 291}` (or equivalent), with `content` empty/null. That is structured tool calling working.
**FAIL:** if `tool_calls` is null/absent and the JSON is dumped in `content` as a string, the parser flag didn't take — verify `--tool-call-parser hermes` is in the running command (`docker compose logs` shows the args at boot) and that the model's chat template actually emits `<tool_call>` tags. Report the raw response.
Also sanity-check throughput:
```bash
# Plain generation latency
curl -s http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"hermes","messages":[{"role":"user","content":"Say hello in one sentence."}]}' \
| python3 -m json.tool
nvidia-smi # confirm VRAM occupancy and that it's the vLLM process
```
---
## 6. Wire into OpenClaw (Pluto's brain)
OpenClaw is model-agnostic and speaks the OpenAI API. Point it at the vLLM endpoint. The vLLM server is on the Windows box at `http://<WINDOWS_LAN_IP>:8000/v1` (or `http://localhost:8000/v1` if OpenClaw runs on the same Windows box; confirm where Pluto's gateway actually runs — it may be on the swarm).
In OpenClaw's model/provider config:
- **Base URL:** `http://<WINDOWS_LAN_IP>:8000/v1`
- **Model name:** `hermes` (matches `--served-model-name`)
- **API key:** any non-empty string (vLLM ignores it by default; use `EMPTY` or `sk-noauth`)
**Networking notes:**
- vLLM in the WSL2 container binds `0.0.0.0:8000`, but WSL2 networking means the port must be reachable from the Windows host and the LAN. WSL2 usually auto-forwards localhost; for LAN access you may need a `netsh interface portproxy` rule on Windows forwarding the host's LAN IP:8000 to the WSL2 IP:8000, plus a Windows Firewall inbound rule for port 8000 **scoped to the LAN subnet only**. Set this up and verify from another machine with `curl http://<WINDOWS_LAN_IP>:8000/v1/models`.
- Do **not** expose port 8000 to the internet. If remote access is wanted later, that goes behind Traefik with forward-auth on the swarm, tunneled to the Windows box — a separate task, not part of this brief.
After wiring, restart the OpenClaw gateway and run a Pluto interaction that should trigger a tool (whatever real tools Pluto has registered) to confirm end-to-end tool execution, not just the synthetic calculator test.
---
## 7. Operational notes / gotchas
- **Display + inference share the 3090.** The desktop's monitor runs off the same card. Under VRAM pressure this can cause display glitches or vLLM OOM. The `0.90` utilization setting accounts for this; if Jeremy ever runs the box headless (RDP only, which the earlier session showed — there was a Remote Display Adapter), more VRAM frees up and utilization can go to 0.95+.
- **vLLM holds VRAM for its lifetime.** It does not unload like Ollama. That's expected and desirable for an always-on Pluto brain, but it means the 3090 is committed — no gaming / other GPU work while it runs.
- **Auto-start on boot:** `restart: unless-stopped` handles container restarts, but WSL2 + Docker Desktop must themselves start on Windows login. For true always-on, enable Docker Desktop "Start on login" and consider WSL2 boot via Task Scheduler, OR (better long-term) move this whole stack to a dedicated headless Linux box with the 3090 if one becomes available. Flag this to Jeremy as a follow-up if always-on reliability matters for VoiceCoach.
- **Context length is capped at 8192 here** for VRAM reasons. Hermes 4.3 supports far more (trained to 512K), but you can't afford the KV cache on 24GB. If Pluto workflows need longer context, that's a hardware conversation (bigger card / second GPU), not a config tweak.
- **Version pinning:** `vllm/vllm-openai:latest` is convenient but pin to a specific tag for production stability once verified working, so an image update doesn't silently break the tool parser.
---
## 8. Report back to Jeremy with
1. `nvidia-smi` output from inside WSL2 (GPU visible y/n).
2. Which model option (A/B/C) you proceeded with and the exact repo ID.
3. The full JSON from the §5 calculator tool-call test (the pass/fail evidence).
4. Measured tokens/sec and VRAM occupancy from `nvidia-smi`.
5. Whether OpenClaw end-to-end tool execution worked with the real Pluto toolset.
6. Any networking steps needed for LAN access (portproxy/firewall rules added).
Do not mark this complete until the §5 tool-call test passes with a structured `tool_calls` array. That single result is the definition of done for the inference layer.