From 7218a11f205c0881a6c502dcb4ab3ca8f04021d3 Mon Sep 17 00:00:00 2001 From: jeremy Date: Tue, 23 Jun 2026 22:04:39 -0500 Subject: [PATCH] docs: LiteLLM proxy design spec Co-Authored-By: Claude Sonnet 4.6 --- .../specs/2026-06-24-litellm-proxy-design.md | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 docs/superpowers/specs/2026-06-24-litellm-proxy-design.md diff --git a/docs/superpowers/specs/2026-06-24-litellm-proxy-design.md b/docs/superpowers/specs/2026-06-24-litellm-proxy-design.md new file mode 100644 index 0000000..526cb89 --- /dev/null +++ b/docs/superpowers/specs/2026-06-24-litellm-proxy-design.md @@ -0,0 +1,100 @@ +# LiteLLM Proxy — Design Spec + +**Date:** 2026-06-24 +**Status:** Approved + +## Overview + +Deploy LiteLLM as a stateless LLM proxy on hf-pdocker-01, exposing a single OpenAI-compatible endpoint on the LAN. Callers pick a model name; LiteLLM routes to the appropriate backend (Anthropic, OpenAI, or local Ollama on the GPU machine). + +## Architecture + +``` +LAN clients (agents, apps, CLI) + │ + ▼ +litellm :4000 (hf-pdocker-01 — 192.168.86.192) + ├── claude-opus-4-8 ──────► Anthropic API (internet) + ├── gpt-5.5 ──────► OpenAI API (internet) + ├── qwen2.5:14b ──────► Ollama :11434 (192.168.86.150) + └── llama3.1:8b ──────► Ollama :11434 (192.168.86.150) +``` + +- **No Traefik route** — LAN-only, port 4000 +- **No authentication** — open access within 192.168.86.x +- **Stateless** — no database, no virtual key management + +## Model List + +| Caller model name | Provider | Backend model ID | +|---|---|---| +| `claude-opus-4-8` | Anthropic | `claude-opus-4-8` | +| `gpt-5.5` | OpenAI | `gpt-5.5` | +| `qwen2.5:14b` | Ollama | `ollama/qwen2.5:14b` | +| `llama3.1:8b` | Ollama | `ollama/llama3.1:8b` | + +## Files + +| File | Location | +|---|---| +| Compose | `compose/hf-pdocker-01/litellm/docker-compose.yml` | +| Env | `compose/hf-pdocker-01/litellm/.env` | +| Config | `/mnt/docker-data/configs/litellm/config.yaml` (NAS) | + +## Compose + +- Image: `ghcr.io/berriai/litellm:main-latest` +- Watchtower disabled +- Port: `4000:4000` +- Config mounted from NAS: `/mnt/docker-data/configs/litellm/config.yaml:/app/config.yaml` +- Credentials via `.env` file + +## Config (config.yaml) + +```yaml +model_list: + - model_name: claude-opus-4-8 + litellm_params: + model: anthropic/claude-opus-4-8 + api_key: os.environ/ANTHROPIC_API_KEY + + - model_name: gpt-5.5 + litellm_params: + model: openai/gpt-5.5 + api_key: os.environ/OPENAI_API_KEY + + - model_name: qwen2.5:14b + litellm_params: + model: ollama/qwen2.5:14b + api_base: http://192.168.86.150:11434 + + - model_name: llama3.1:8b + litellm_params: + model: ollama/llama3.1:8b + api_base: http://192.168.86.150:11434 + +litellm_settings: + drop_params: true +``` + +## Credentials + +- `ANTHROPIC_API_KEY` — copied from existing ai-agents `.env` on NAS +- `OPENAI_API_KEY` — placeholder; user must supply before first use +- Ollama — no key required + +## Deployment Steps + +1. Create config dir on NAS: `/mnt/docker-data/configs/litellm/` +2. Write `config.yaml` to NAS +3. Write `docker-compose.yml` and `.env` to NAS and repo +4. SSH to hf-pdocker-01, `docker compose up -d` +5. Verify: `curl http://192.168.86.192:4000/health` + +## Usage + +```bash +curl http://192.168.86.192:4000/v1/chat/completions \ + -H "Content-Type: application/json" \ + -d '{"model": "claude-opus-4-8", "messages": [{"role": "user", "content": "Hello"}]}' +```