Multi-tenant LLM Gateway on APISIX

Apache APISIX gateway for shared LLM traffic with virtual key sharding, spend limits, PII redaction, and built-in safety and moderation pipelines.

Cloud backends run through ai-proxy or relay configuration, including OpenAI, Anthropic, Gemini, Bedrock, and others, with usage, cost, and health tracked in ClickHouse and Grafana.

This repo ships sample routes to OpenCode and llamafile.

Default deployment routes cloud traffic to OpenCode Go (opencode.ai). The gateway itself is provider-agnostic: add relay routes or swap to ai-proxy / ai-proxy-multi for any OpenAI-compatible backend (see Supported Providers).

Full technical reference: docs/architecture/README.md

Table of Contents

• Quick Start
• Architecture
• Supported Providers
• Features
• Work in progress
• Plugins
• Key Management
• Configuration
• opencode Integration
• Testing
• Make Targets
• Documentation
• License

Quick Start

# 1. Install podman-compose and build images
make install

# 2. Start the gateway stack (APISIX + etcd + ClickHouse + Vector + OpenBao + Prometheus + Grafana)
make dev-start

# 3. Send a request through the gateway
KEY="$GATEWAY_API_KEY"  # vgw-gateway-key from .env (provisioned in OpenBao on start)
curl -s http://localhost:9080/opencode_federated/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"minimax-m3","messages":[{"role":"user","content":"Say hello"}]}'

Ports: 9080 (gateway), 9180 (APISIX Admin API + /ui/), 8123 (ClickHouse), 9100 (APISIX Prometheus metrics), 8201 (OpenBao), 3030 (Grafana), 9092 (Prometheus).

Prerequisites

• Podman 5.x
• Ansible 2.21+
• curl, jq, openssl, xxd (used by tests and key scripts)
• uv (for .venv setup)
• A .env file with ADMIN_KEY, OPENCODE_API_KEY, GATEWAY_API_KEY, OPENBAO_TOKEN (see .env.example, gitignored)

Run make init to check all system dependencies and print install instructions for any that are missing.

Architecture

Legend (all diagrams in this doc): solid arrows = request/response data path; dashed arrows = config, key lookup, or read-only observability.

System context

flowchart TB
    Clients[Clients]
    Gateway["LLM Gateway<br/>APISIX :9080"]
    Cloud["Cloud LLM APIs<br/>OpenAI · Anthropic · xAI · ..."]
    Local["Local inference<br/>llamafile · vLLM · ..."]
    OpenBao[(OpenBao)]
    Store[(ClickHouse)]

    Clients -->|HTTPS| Gateway
    Gateway -->|cloud relays| Cloud
    Gateway -->|local relay| Local
    Gateway -.->|vgw-* keys| OpenBao
    Gateway -.->|usage and request logs| Store

The gateway is provider-agnostic: clients hit APISIX, which relays to cloud or local LLM APIs, resolves virtual keys via OpenBao, and persists usage data. Deeper flows are diagrammed in the section that owns each concern: Plugins (request path), Configuration (telemetry, metrics, route config), Key Management (auth).

Each new provider is a relay route + upstream node (or ai-proxy / ai-proxy-multi; see docs/specifications/SPEC-ENTERPRISE-AUTH.md and Supported Providers). Diagram authoring rules: ../CI/workflows/WORKFLOW-CREATING-DIAGRAMS.md.

Sample deployments in this repo

In this sample, OpenCode Go exposes 20+ models (MiniMax, Kimi, GLM, DeepSeek, Qwen, MiMo, HY3). Swap the upstream node in apisix.yaml.j2 to point at any other compatible API. Additional providers = new relay route + upstream node; see docs/specifications/SPEC-PROVIDER-XAI.md for the xAI Grok draft spec and docs/specifications/SPEC-PROVIDER-KIMI.md for the Moonshot Kimi spec.

Supported Providers

APISIX's built-in ai-proxy and ai-proxy-multi plugins support the following LLM provider backends. Swap the plain upstream proxy in conf/apisix.yaml for ai-proxy (single provider) or ai-proxy-multi (load balancing, retries, health checks across multiple providers).

ai-proxy-multi adds: load balancing across instances, automatic retries on failure, health checks, and provider-level routing rules.

Features

Work in progress

Built-in safety and moderation (gateway plugins)

Status: WIP. Target: APISIX plugins on the federated request path that score assistant behavior and enforce moderation policy before or after upstream relay, alongside existing redact and key-resolver policy plugins.

POC benchmark (WORKSPACE-VM): the llamafile transcript classifier benchmark replays OpenCode SQLite sessions turn-by-turn against a local llamafile server. Each step sends a rolling 32K transcript window to a small classifier model and expects YAML-only scores across seven behavioral categories (instruction adherence, task understanding, sincerity, malicious intent, competence, safety boundaries, clarity). KV cache reuse via pinned id_slot and cache_prompt keeps incremental moderation affordable at gateway latency budgets.

Run the POC from the WORKSPACE-VM repo root (llamafile server must be up):

make -f Makefile.llamafile benchmark-llamafile-transcript-classifier

Config and categories: benchmarks/llamafile/transcript_classifier/benchmark.yaml. This repo already exposes llamafile via relay-llamafile (/llamafile/*); the WIP work wires the classifier/moderator call pattern into APISIX as first-class plugins with ClickHouse audit rows alongside usage_log.

Plugins

Request path (one federated cloud request)

flowchart TB
    Client[Client]
    Route["/opencode_federated/*"]
    Policy[Policy plugins]
    Proxy[Proxy plugins]
    Upstream[Cloud upstream]
    OpenBao[(OpenBao)]

    Client --> Route --> Policy --> Proxy --> Upstream
    OpenBao -.->|key lookup| Policy

Policy = key-resolver, key-meta, redact, limit-count (access phase). Proxy = proxy-rewrite, upstream proxy, proxy-buffering. Telemetry plugins run after the upstream responds; see ClickHouse Tables.

/opencode/* skips key-resolver; /llamafile/* skips auth and targets a local upstream (see sample deployments).

Eight plugins on the passthrough and llamafile routes, nine on the federated route (federated adds key-resolver), ordered by Nginx phase priority:

• proxy-rewrite (N/A, Built-in, rewrite) : Strips route prefix; opencode relays → /zen/go/*, llamafile → upstream root
• key-resolver (2555, Custom Lua, access, federated only) : Resolve vgw-* keys via OpenBao; pass through others
• key-meta (2530, Custom Lua, access) : Compute key hash for per-key scoping (X-Key-Hash)
• redact (2500, Custom Lua, access/header_filter/body_filter/log) : PII anonymization + re-hydration
• sse-usage (2400, Custom Lua, header_filter/body_filter/log) : Extract token usage; increment budget counter
• limit-count (2002, Built-in, access) : Per-key RPM; federated route uses variable limits from OpenBao headers
• http-logger (410, Built-in, log) : Send req/resp metadata to Vector
• proxy-buffering (300, Built-in, filter) : Disable buffering for SSE
• prometheus (N/A, Built-in, log) : Export metrics at :9100

Extract-Testable-Core Pattern

Each custom plugin is split into two files:

• *_lib.lua : Pure logic module, requireable, unit-testable (deps: cjson, ngx.re only)
• *.lua : APISIX adapter: lifecycle phases, ctx, shared dict (deps: Full APISIX API)

Key Management

flowchart TB
    REQ["Authorization: Bearer token"] --> CHECK{"Starts with vgw-?"}
    CHECK -->|"YES"| OPENBAO["Lookup in OpenBao"]
    OPENBAO -->|"Found + active"| INJECT["Inject upstream provider key"]
    OPENBAO -->|"Not found"| R401["401 invalid key"]
    OPENBAO -->|"Revoked"| R401R["401 key revoked"]
    OPENBAO -->|"Unreachable"| R503["503 key store unreachable"]
    CHECK -->|"NO"| PASS["Pass through provider key"]
    INJECT -->|"federated route"| UPSTREAM["Configured upstream provider"]
    PASS -->|"passthrough route"| UPSTREAM

Applies to /opencode/* and /opencode_federated/* only; /llamafile/* has no Authorization flow.

Key Modes

1. Virtual keys (vgw-*): Used on the /opencode_federated/* and /kimi-federated/* routes. Stored in OpenBao (production file-storage mode with persistent volumes). Resolved to an upstream provider API key. Can be revoked, rate-limited per tenant, audited. Cached in key_cache shared dict (5s TTL in dev, 300s in prod).

2. Direct keys (any non-vgw- prefix, e.g. sk-*): Used on the /opencode/* and /kimi-key/* routes. Passed through to upstream as-is. No OpenBao lookup. Users bring their own upstream provider API keys.

3. Upstream key pools: Named pools of upstream API keys shared by one or more virtual keys. The key-resolver plugin selects keys sticky-style and rotates on upstream quota/rate-limit responses (429 parks a key in cooldown, 402/403 hard-disables it in OpenBao). Create and attach pools via make pool-key and make issue-key POOL=.... Details: docs/runbooks/RUNBOOK-KEYS.md + docs/architecture/KEY-MANAGEMENT.md.

Commands

make issue-key                              # Create vgw-<random hex> key
make issue-key KEY_ID=my-key TENANT_ID=acme USER_ID=alice
make issue-key KEY_ID=my-key POOL=kimi      # Attach key to upstream pool
make list-keys                              # List all keys with metadata
make revoke-key KEY_ID=vgw-abc123           # Revoke (record preserved)
make pool-key ARGS='list'                   # List upstream key pools
make pool-key ARGS='create kimi'            # Create a new pool
make pool-key ARGS='add kimi k1 sk-...'     # Add a key to a pool

Configuration

Routes and config (control plane)

flowchart TB
    J2[apisix.yaml.j2]
    YAML[apisix.yaml]
    Seed[seed-routes.sh]
    Etcd[(etcd)]
    Routes[APISIX routes]
    Admin["Admin API :9180"]

    J2 --> YAML --> Seed --> Etcd --> Routes
    Admin -.->|route CRUD| Etcd

Traditional/etcd mode: routes live in etcd, seeded from the rendered conf/apisix.yaml on stack start. Admin API and built-in dashboard: http://localhost:9180/ui/.

Key Files

• conf/config.yaml: APISIX traditional/etcd mode: plugin list, shared dicts, env vars, Admin API, Prometheus port
• conf/apisix.yaml: Committed route render (3 routes); drift-checked against conf/apisix.yaml.j2
• conf/apisix.yaml.j2: Jinja2 route template rendered at deploy from .env
• res/scripts/seed-routes.sh: Seeds etcd from rendered apisix.yaml on stack start
• conf/openbao.hcl: OpenBao production config (file-storage backend)
• conf/prometheus.yml: Prometheus scrape config (APISIX :9100)
• conf/grafana/: Grafana datasources + 3 provisioned dashboards
• conf/redact-patterns.json: PII detection: 6 regex patterns + 2 dictionary categories
• conf/clickhouse-init.sql: Base schema; incremental changes via conf/migrations/
• conf/vector.toml: Vector pipeline: HTTP source, VRL remap (parse_json for model extraction), ClickHouse sink
• res/docker/docker-compose.yml: 8 services: apisix, etcd, clickhouse, migrate, vector, openbao, prometheus, grafana
• res/docker/Dockerfile.apisix: Custom APISIX image: Lua plugins + config copied in
• res/docker/Dockerfile.openbao: Custom OpenBao image (production file-storage)
• res/docker/openbao-entrypoint.sh: OpenBao auto-init, auto-unseal, gateway key provisioning (data persists via openbao-data named volume)
• .env: Secrets: ADMIN_KEY, OPENCODE_API_KEY, GATEWAY_API_KEY, OPENBAO_TOKEN

Environment Variables

ClickHouse Tables

Telemetry runs in response/log phases after the upstream returns:

flowchart TB
    Response[Upstream response]
    Tele[Telemetry plugins]
    CH[(ClickHouse)]
    Vector[Vector]

    Response --> Tele
    Tele -->|usage_log| CH
    Tele -->|request_log| Vector
    Vector --> CH

sse-usage writes usage_log directly; http-logger ships full request/response metadata to Vector, which inserts request_log.

Grafana Dashboards

flowchart TB
    Plugin[prometheus plugin]
    Export["APISIX :9100"]
    Prom[(Prometheus)]
    Grafana[Grafana]
    CH[(ClickHouse)]

    Plugin --> Export
    Prom -.->|scrape| Export
    Grafana -.->|PromQL| Prom
    Grafana -.->|SQL queries| CH

The prometheus plugin exports request metrics at :9100. Prometheus scrapes every 15s (conf/prometheus.yml). Grafana uses Prometheus for ops panels (latency, error rate) and ClickHouse for cost and usage. Grafana only queries data; it does not write.

Three provisioned dashboards (default: now-7d lookback, 5s refresh):

The leaderboard shows top clients (p20) and top models (p21) by cost and tokens. After editing dashboard JSON, run make dev-restart-grafana to reload provisioning. See docs/specifications/SPEC-DASHBOARD.md.

opencode Integration

The gateway registers as workspace-gw-private (virtual key), workspace-gw-own (own key), workspace-gw-llamafile (local LLM), and three Moonshot Kimi providers — workspace-gw-kimi-oauth (OAuth device flow), workspace-gw-kimi-private (virtual key), and workspace-gw-kimi-own (own key) — as custom providers in opencode.

# Sync all models from gateway into opencode config
make sync-models

This fetches /opencode_federated/v1/models from the gateway using the virtual gateway key, enriches each model with canonical metadata (name, context limit, capabilities, cost, modalities) from models.dev, and also fetches /llamafile/v1/models for the local llamafile upstream. It writes provider entries into ~/.config/opencode/opencode.jsonc for the OpenCode and Kimi access modes, plus llamafile:

• workspace-gw-private: virtual-key mode (apiKey = vgw-gateway-key)
• workspace-gw-own: own-key passthrough (no apiKey, client provides key)
• workspace-gw-llamafile: no-auth local LLM (VM-hosted llamafile, no apiKey)
• workspace-gw-kimi-oauth: OAuth device-flow (gateway-managed Kimi token)
• workspace-gw-kimi-private: virtual-key mode for Kimi (vgw-*)
• workspace-gw-kimi-own: own Moonshot key passthrough for Kimi

For the OAuth providers, use the login script in docs/runbooks/RUNBOOK-CLIENT-LOGIN.md which starts the device flow and prints the verification URL.

The first two providers receive the full enriched model catalog so opencode does not drop them (opencode deletes providers with zero models). The llamafile provider receives the model list from /llamafile/v1/models (or a default model id if the llamafile server is not running). MiniCPM5 uses context 131072 (scaled to 104857 at 80%) with tool_call: true. The script runs automatically on make dev-start and make dev-restart via the Ansible playbook.

Context limits are scaled by CONTEXT_LIMIT_PCT (default 80) from .env, so e.g. CONTEXT_LIMIT_PCT=80 reduces a 200000-token context to 160000. An absolute ceiling CONTEXT_LIMIT_CEILING (default 128000) is then applied: any scaled value exceeding the ceiling is clamped to it. Set to 0 to disable.

Result in opencode config:

{
  "provider": {
    "workspace-gw-private": {
      "api": "http://localhost:9080/opencode_federated/v1",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:9080/opencode_federated/v1",
        "apiKey": "vgw-gateway-key",
        "headers": { "X-Tenant-ID": "default", "X-User-ID": "agent" }
      },
      "models": {
        "minimax-m3": {
          "name": "MiniMax M3",
          "family": "minimax",
          "release_date": "2026-06-01",
          "attachment": true,
          "reasoning": true,
          "temperature": true,
          "tool_call": true,
          "cost": { "input": 15, "output": 75, "cache_read": 1.5, "cache_write": 18.75 },
          "limit": { "context": 160000, "output": 24000 },
          "modalities": { "input": ["text", "image", "pdf"], "output": ["text"] },
          "status": "active"
        }
      }
    },
    "workspace-gw-own": {
      "api": "http://localhost:9080/opencode/v1",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:9080/opencode/v1",
        "headers": { "X-Tenant-ID": "default", "X-User-ID": "agent" }
      },
      "models": { "...": "same enriched models as workspace-gw-private" }
    }
  }
}

Testing

make test          # Run all stages (excludes live upstream API tests)
make test-live     # Run all stages including live upstream API tests
make dev-test      # Same as test, via Ansible

1. Lua unit tests via resty CLI inside the APISIX container
2. Config validation: 14 scripts (YAML, SQL, TOML, JSON, dashboard structure, migrations)
3. Reconciler static analysis: syntax, strict mode, error handling
4. Integration: black-box HTTP against the running stack (llamafile e2e, event_id alignment, data flow, cost e2e, Grafana panel checks)
5. CI hook verification: pre-commit and pre-push hooks present and wired
6. E2E: real Go API calls (gated behind RUN_LIVE_API_TESTS=1)

See docs/testplans/TEST-PLAN.md for the full strategy.

Make Targets

Dev Lifecycle (Ansible-managed)

Key Management

Quality Gates

Documentation

• ../CI/workflows/WORKFLOW-CREATING-DIAGRAMS.md : How we author and review architecture diagrams
• docs/README.md : Documentation hub: full tree and reading order
• docs/architecture/README.md : Architecture hub: components, plugins, data flows, schema
• docs/requirements/ : Functional/non-functional requirements (REQ-*, RFC 2119)
• docs/specifications/ : Implementation specifications (SPEC-*), one per REQ
• docs/runbooks/RUNBOOK-DEPLOYMENT.md : Deployment and operations runbook
• docs/runbooks/RUNBOOK-KEYS.md : Gateway key lifecycle (issue/list/revoke) and upstream key pool management
• docs/runbooks/RUNBOOK-CLIENT-LOGIN.md : Client provider login script usage
• docs/testplans/TEST-PLAN.md : Testing strategy with extract-testable-core pattern
• docs/reference/OPENCODE-SERVER-API.md : Upstream opencode server API reference
• docs/proposals/ADR-001-APISIX-PIVOT.md : Architecture rationale, Kong-to-APISIX pivot

License

• Apache APISIX 3.17.0: Apache 2.0
• OpenBao 2.4.4: MPL 2.0
• ClickHouse 24.8: Apache 2.0
• Vector 0.40: MPL 2.0
• Prometheus v3.13.1: Apache 2.0
• Grafana 13.0.2: AGPLv3