Files

T

jeremy 7218a11f20 docs: LiteLLM proxy design spec

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-06-23 22:04:39 -05:00

2.8 KiB

Raw Blame History

LiteLLM Proxy — Design Spec

Date: 2026-06-24
Status: Approved

Overview

Deploy LiteLLM as a stateless LLM proxy on hf-pdocker-01, exposing a single OpenAI-compatible endpoint on the LAN. Callers pick a model name; LiteLLM routes to the appropriate backend (Anthropic, OpenAI, or local Ollama on the GPU machine).

Architecture

LAN clients (agents, apps, CLI)
        │
        ▼
litellm :4000  (hf-pdocker-01 — 192.168.86.192)
   ├── claude-opus-4-8  ──────► Anthropic API (internet)
   ├── gpt-5.5          ──────► OpenAI API (internet)
   ├── qwen2.5:14b      ──────► Ollama :11434 (192.168.86.150)
   └── llama3.1:8b      ──────► Ollama :11434 (192.168.86.150)

No Traefik route — LAN-only, port 4000
No authentication — open access within 192.168.86.x
Stateless — no database, no virtual key management

Model List

Caller model name	Provider	Backend model ID
`claude-opus-4-8`	Anthropic	`claude-opus-4-8`
`gpt-5.5`	OpenAI	`gpt-5.5`
`qwen2.5:14b`	Ollama	`ollama/qwen2.5:14b`
`llama3.1:8b`	Ollama	`ollama/llama3.1:8b`

Files

File	Location
Compose	`compose/hf-pdocker-01/litellm/docker-compose.yml`
Env	`compose/hf-pdocker-01/litellm/.env`
Config	`/mnt/docker-data/configs/litellm/config.yaml` (NAS)

Compose

Image: ghcr.io/berriai/litellm:main-latest
Watchtower disabled
Port: 4000:4000
Config mounted from NAS: /mnt/docker-data/configs/litellm/config.yaml:/app/config.yaml
Credentials via .env file

Config (config.yaml)

model_list:
  - model_name: claude-opus-4-8
    litellm_params:
      model: anthropic/claude-opus-4-8
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

  - model_name: qwen2.5:14b
    litellm_params:
      model: ollama/qwen2.5:14b
      api_base: http://192.168.86.150:11434

  - model_name: llama3.1:8b
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: http://192.168.86.150:11434

litellm_settings:
  drop_params: true

Credentials

ANTHROPIC_API_KEY — copied from existing ai-agents .env on NAS
OPENAI_API_KEY — placeholder; user must supply before first use
Ollama — no key required

Deployment Steps

Create config dir on NAS: /mnt/docker-data/configs/litellm/
Write config.yaml to NAS
Write docker-compose.yml and .env to NAS and repo
SSH to hf-pdocker-01, docker compose up -d
Verify: curl http://192.168.86.192:4000/health

Usage

curl http://192.168.86.192:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "claude-opus-4-8", "messages": [{"role": "user", "content": "Hello"}]}'

2.8 KiB Raw Blame History