101 lines
2.8 KiB
Markdown
101 lines
2.8 KiB
Markdown
# LiteLLM Proxy — Design Spec
|
|
|
|
**Date:** 2026-06-24
|
|
**Status:** Approved
|
|
|
|
## Overview
|
|
|
|
Deploy LiteLLM as a stateless LLM proxy on hf-pdocker-01, exposing a single OpenAI-compatible endpoint on the LAN. Callers pick a model name; LiteLLM routes to the appropriate backend (Anthropic, OpenAI, or local Ollama on the GPU machine).
|
|
|
|
## Architecture
|
|
|
|
```
|
|
LAN clients (agents, apps, CLI)
|
|
│
|
|
▼
|
|
litellm :4000 (hf-pdocker-01 — 192.168.86.192)
|
|
├── claude-opus-4-8 ──────► Anthropic API (internet)
|
|
├── gpt-5.5 ──────► OpenAI API (internet)
|
|
├── qwen2.5:14b ──────► Ollama :11434 (192.168.86.150)
|
|
└── llama3.1:8b ──────► Ollama :11434 (192.168.86.150)
|
|
```
|
|
|
|
- **No Traefik route** — LAN-only, port 4000
|
|
- **No authentication** — open access within 192.168.86.x
|
|
- **Stateless** — no database, no virtual key management
|
|
|
|
## Model List
|
|
|
|
| Caller model name | Provider | Backend model ID |
|
|
|---|---|---|
|
|
| `claude-opus-4-8` | Anthropic | `claude-opus-4-8` |
|
|
| `gpt-5.5` | OpenAI | `gpt-5.5` |
|
|
| `qwen2.5:14b` | Ollama | `ollama/qwen2.5:14b` |
|
|
| `llama3.1:8b` | Ollama | `ollama/llama3.1:8b` |
|
|
|
|
## Files
|
|
|
|
| File | Location |
|
|
|---|---|
|
|
| Compose | `compose/hf-pdocker-01/litellm/docker-compose.yml` |
|
|
| Env | `compose/hf-pdocker-01/litellm/.env` |
|
|
| Config | `/mnt/docker-data/configs/litellm/config.yaml` (NAS) |
|
|
|
|
## Compose
|
|
|
|
- Image: `ghcr.io/berriai/litellm:main-latest`
|
|
- Watchtower disabled
|
|
- Port: `4000:4000`
|
|
- Config mounted from NAS: `/mnt/docker-data/configs/litellm/config.yaml:/app/config.yaml`
|
|
- Credentials via `.env` file
|
|
|
|
## Config (config.yaml)
|
|
|
|
```yaml
|
|
model_list:
|
|
- model_name: claude-opus-4-8
|
|
litellm_params:
|
|
model: anthropic/claude-opus-4-8
|
|
api_key: os.environ/ANTHROPIC_API_KEY
|
|
|
|
- model_name: gpt-5.5
|
|
litellm_params:
|
|
model: openai/gpt-5.5
|
|
api_key: os.environ/OPENAI_API_KEY
|
|
|
|
- model_name: qwen2.5:14b
|
|
litellm_params:
|
|
model: ollama/qwen2.5:14b
|
|
api_base: http://192.168.86.150:11434
|
|
|
|
- model_name: llama3.1:8b
|
|
litellm_params:
|
|
model: ollama/llama3.1:8b
|
|
api_base: http://192.168.86.150:11434
|
|
|
|
litellm_settings:
|
|
drop_params: true
|
|
```
|
|
|
|
## Credentials
|
|
|
|
- `ANTHROPIC_API_KEY` — copied from existing ai-agents `.env` on NAS
|
|
- `OPENAI_API_KEY` — placeholder; user must supply before first use
|
|
- Ollama — no key required
|
|
|
|
## Deployment Steps
|
|
|
|
1. Create config dir on NAS: `/mnt/docker-data/configs/litellm/`
|
|
2. Write `config.yaml` to NAS
|
|
3. Write `docker-compose.yml` and `.env` to NAS and repo
|
|
4. SSH to hf-pdocker-01, `docker compose up -d`
|
|
5. Verify: `curl http://192.168.86.192:4000/health`
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
curl http://192.168.86.192:4000/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"model": "claude-opus-4-8", "messages": [{"role": "user", "content": "Hello"}]}'
|
|
```
|