# LiteLLM Proxy — Design Spec

**Date:** 2026-06-24  
**Status:** Approved

## Overview

Deploy LiteLLM as a stateless LLM proxy on hf-pdocker-01, exposing a single OpenAI-compatible endpoint on the LAN. Callers pick a model name; LiteLLM routes to the appropriate backend (Anthropic, OpenAI, or local Ollama on the GPU machine).

## Architecture

```
LAN clients (agents, apps, CLI)
        │
        ▼
litellm :4000  (hf-pdocker-01 — 192.168.86.192)
   ├── claude-opus-4-8  ──────► Anthropic API (internet)
   ├── gpt-5.5          ──────► OpenAI API (internet)
   ├── qwen2.5:14b      ──────► Ollama :11434 (192.168.86.150)
   └── llama3.1:8b      ──────► Ollama :11434 (192.168.86.150)
```

- **No Traefik route** — LAN-only, port 4000
- **No authentication** — open access within 192.168.86.x
- **Stateless** — no database, no virtual key management

## Model List

| Caller model name | Provider | Backend model ID |
|---|---|---|
| `claude-opus-4-8` | Anthropic | `claude-opus-4-8` |
| `gpt-5.5` | OpenAI | `gpt-5.5` |
| `qwen2.5:14b` | Ollama | `ollama/qwen2.5:14b` |
| `llama3.1:8b` | Ollama | `ollama/llama3.1:8b` |

## Files

| File | Location |
|---|---|
| Compose | `compose/hf-pdocker-01/litellm/docker-compose.yml` |
| Env | `compose/hf-pdocker-01/litellm/.env` |
| Config | `/mnt/docker-data/configs/litellm/config.yaml` (NAS) |

## Compose

- Image: `ghcr.io/berriai/litellm:main-latest`
- Watchtower disabled
- Port: `4000:4000`
- Config mounted from NAS: `/mnt/docker-data/configs/litellm/config.yaml:/app/config.yaml`
- Credentials via `.env` file

## Config (config.yaml)

```yaml
model_list:
  - model_name: claude-opus-4-8
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

  - model_name: qwen2.5:14b
    litellm_params:
      model: ollama/qwen2.5:14b
      api_base: http://192.168.86.150:11434

  - model_name: llama3.1:8b
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://192.168.86.150:11434

litellm_settings:
  drop_params: true
```

## Credentials

- `ANTHROPIC_API_KEY` — copied from existing ai-agents `.env` on NAS
- `OPENAI_API_KEY` — placeholder; user must supply before first use
- Ollama — no key required

## Deployment Steps

1. Create config dir on NAS: `/mnt/docker-data/configs/litellm/`
2. Write `config.yaml` to NAS
3. Write `docker-compose.yml` and `.env` to NAS and repo
4. SSH to hf-pdocker-01, `docker compose up -d`
5. Verify: `curl http://192.168.86.192:4000/health`

## Usage

```bash
curl http://192.168.86.192:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-opus-4-8", "messages": [{"role": "user", "content": "Hello"}]}'
```