Multi-tenant LLM Gateway on APISIX

WORKSPACE-GATEWAY relays shared LLM traffic through Apache APISIX with per-tenant virtual keys, spend limits, and PII redaction. It speaks each provider's native protocol, with usage, cost, and health tracked in ClickHouse and Grafana; this repo ships sample routes to OpenCode Go and Zen, Kimi, Z.ai, Anthropic, OpenAI, Alibaba Token Plan, and a local llamafile.

Table of Contents

• Quick Start
• Architecture
• Features
• Work in progress
• Plugins
• Key Management
• Configuration
• opencode Integration
• Testing
• Make Targets
• License

Quick Start

# 1. Install podman-compose and build images
make install

# 2. Start the gateway stack (APISIX + etcd + ClickHouse + Vector + OpenBao + Prometheus + Grafana)
make gw-start

# 3. Send a request through the gateway
KEY="$GATEWAY_API_KEY"  # vgw-gateway-key from .env (provisioned in OpenBao on start)
curl -s http://localhost:9080/opencode_federated/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"minimax-m3","messages":[{"role":"user","content":"Say hello"}]}'

Host port surface:

etcd, OpenBao, Vector, Prometheus, the APISIX Admin API and metrics, and the ClickHouse native port publish no host ports; reach them with podman exec.

Prerequisites

• Podman 5.x
• Ansible 2.21+
• curl, jq, openssl, xxd (used by tests and key scripts)
• uv (for .venv setup)
• A .env file with ADMIN_KEY, OPENCODE_API_KEY, GATEWAY_API_KEY, OPENBAO_TOKEN, GRAFANA_ADMIN_PASSWORD, the ClickHouse credential set, and the etcd credential set (see .env.example, gitignored)

Run make init to check all system dependencies and print install instructions for any that are missing.

Architecture

Legend (all diagrams in this doc): solid arrows = request/response data path; dashed arrows = config, key lookup, or read-only observability.

System context

flowchart TB
    Clients[Clients]
    Gateway["LLM Gateway<br/>APISIX :9080"]
    Cloud["Cloud LLM APIs<br/>OpenAI · Anthropic · xAI · ..."]
    Local["Local inference<br/>llamafile · vLLM · ..."]
    OpenBao[(OpenBao)]
    Store[(ClickHouse)]

    Clients -->|HTTPS| Gateway
    Gateway -->|cloud relays| Cloud
    Gateway -->|local relay| Local
    Gateway -.->|vgw-* keys| OpenBao
    Gateway -.->|usage and request logs| Store

Clients hit APISIX, which relays to cloud or local LLM APIs, resolves virtual keys via OpenBao, and persists usage data. Deeper flows are diagrammed in the section that owns each concern: Plugins (request path), Configuration (telemetry, metrics, route config), Key Management (auth).

Sample deployments in this repo

All 18 relay routes as registered in APISIX (definitions in conf/apisix.yaml, provider config in conf/providers/):

In this sample, OpenCode Go exposes 20+ models (MiniMax, Kimi, GLM, DeepSeek, Qwen, MiMo, HY3) and OpenCode Zen serves the free/Zen model set (*-free + pay-as-you-go) via the /opencode_zen/* relay. Swap the upstream node in apisix.yaml.j2 to point at any other compatible API. Upstream API-key quota exhaustion is handled by upstream key pools (see Key Management).

Features

• Native-protocol passthrough (no protocol translation): per-route upstream + proxy-rewrite; ai-proxy deliberately absent (Config).
• PII redaction (on-the-fly sensitive data anonymisation) + re-hydration: redact, regex + dictionary + Luhn, pure Lua (Custom).
• Virtual key management: key-resolver, OpenBao KVv2 (persistent file-storage) with a shared dict cache (Custom).
• Direct key pass-through: key-resolver, non-vgw- keys forwarded as-is (Custom).
• Upstream key pool rotation: key-resolver + upstream_pool_lib.lua, sticky selection with auto-rotate on 429/402/403 (Custom).
• OAuth device/browser flows (any provider): provider-oauth + oauth_device / oauth_jwt / oauth_store, protocol engines with per-route config, OpenBao session storage, and transparent refresh (Custom).
• SSE token extraction: sse-usage, buffers SSE, extracts usage, writes ClickHouse (Custom).
• Per-key rate limiting (RPM): limit-count + key-meta (Built-in + custom Lua).
• Per-key token/cost budget: key-resolver + sse-usage + ngx.shared (Custom Lua).
• Request/response logging: http-logger to Vector to ClickHouse (Built-in).
• Prometheus metrics: prometheus at :9100 (Built-in).
• SSE streaming support: proxy-buffering disabled per-route (Config).
• Grafana dashboards (5): Cost & Usage, Ops & Health, Cost Leaderboard, Model Experience, Model Performance, 90d lookback with 5s refresh (Config).
• Billing-grade schema: ClickHouse Decimal64(6), tiered retention (archive volume, no deletion), LowCardinality keys (SQL).

Work in progress

Built-in safety and moderation (gateway plugins)

Status: WIP. Target: APISIX plugins on the federated request path that score assistant behavior and enforce moderation policy before or after upstream relay, alongside existing redact and key-resolver policy plugins.

Semantic response cache

Status: WIP. A semantic-cache plugin (pgvector-backed response reuse) is not attached to any route; it needs a pgvector store added to the stack.

Plugins

Request path (one federated cloud request)

flowchart TB
    Client[Client]
    Route["/opencode_federated/*"]
    Policy[Policy plugins]
    Proxy[Proxy plugins]
    Upstream[Cloud upstream]
    OpenBao[(OpenBao)]

    Client --> Route --> Policy --> Proxy --> Upstream
    OpenBao -.->|key lookup| Policy

Policy = key-resolver, key-meta, redact, limit-count (access phase). Proxy = proxy-rewrite, upstream proxy, proxy-buffering. Telemetry plugins run after the upstream responds; see ClickHouse Tables.

/opencode/* skips key-resolver; /llamafile/* skips auth and targets a local upstream (see sample deployments).

Plugins on the request path, in Nginx phase-priority order (provider-oauth replaces key-resolver on the kimi routes):

• proxy-rewrite (N/A, Built-in, rewrite) : Strips route prefix; opencode relays → /zen/go/*, opencode zen relay → /zen/*, llamafile → upstream root
• provider-oauth (2560, Custom Lua, access, kimi routes) : Upstream device/browser OAuth with transparent token refresh
• key-resolver (2555, Custom Lua, access, federated only) : Resolve vgw-* keys via OpenBao; pass through others
• key-meta (2530, Custom Lua, access) : Compute key hash for per-key scoping (X-Key-Hash)
• redact (2500, Custom Lua, access/header_filter/body_filter/log) : PII anonymization + re-hydration
• sse-usage (2400, Custom Lua, header_filter/body_filter/log) : Extract token usage; increment budget counter
• limit-count (2002, Built-in, access) : Per-key RPM; federated route uses variable limits from OpenBao headers
• request-id (N/A, Built-in, rewrite/log) : Add X-Request-Id and echo it in the response
• http-logger (410, Built-in, log) : Send req/resp metadata to Vector
• proxy-buffering (300, Built-in, filter) : Disable buffering for SSE
• prometheus (N/A, Built-in, log) : Export metrics at :9100

Extract-Testable-Core Pattern

Each custom plugin is split into two files:

• *_lib.lua : Pure logic module, requireable, unit-testable (deps: cjson, ngx.re only)
• *.lua : APISIX adapter: lifecycle phases, ctx, shared dict (deps: Full APISIX API)

Key Management

flowchart TB
    REQ["Authorization: Bearer token"] --> CHECK{"Starts with vgw-?"}
    CHECK -->|"YES"| OPENBAO["Lookup in OpenBao"]
    OPENBAO -->|"Found + active"| INJECT["Inject upstream provider key"]
    OPENBAO -->|"Not found"| R401["401 invalid key"]
    OPENBAO -->|"Revoked"| R401R["401 key revoked"]
    OPENBAO -->|"Unreachable"| R503["503 key store unreachable"]
    CHECK -->|"NO"| PASS["Pass through provider key"]
    INJECT -->|"federated route"| UPSTREAM["Configured upstream provider"]
    PASS -->|"passthrough route"| UPSTREAM

Applies to /opencode/* and /opencode_federated/* only; /llamafile/* has no Authorization flow.

Key Modes

1. Virtual keys (vgw-*): used on relay-opencode-federated, relay-kimi-federated, and relay-kimi-federated-v1. Stored in OpenBao KV v2 and resolved at request time to an upstream provider key. Revocable, rate-limited per tenant, audited. Cached in the key_cache shared dict (5s TTL in dev, 300s in prod).

2. Direct keys (any non-vgw- prefix, e.g. sk-*): used on the /opencode/*, /kimi-key/*, and /zai-key/* routes. Passed through to upstream as-is. No OpenBao lookup. Users bring their own provider API keys.

3. Upstream key pools: named sets of upstream API keys shared by one or more virtual keys. key-resolver selects pool keys sticky-style and rotates on upstream quota/rate-limit responses (429 parks a key in cooldown, 402/403 hard-disables it in OpenBao).

Mapping precedence at request time: a non-empty upstream_pool wins, then the key's own upstream_key, then the gateway-wide OPENCODE_API_KEY.

Commands

Every key operation goes through one dispatcher, make key ARGS='...' (wrapper: res/scripts/gateway-key.sh). The older single-purpose targets (make issue-key, make list-keys, make revoke-key, make pool-key) remain, but they expose fewer flags: issue rate-limit and budget options are reachable only through make key or the script directly.

make key ARGS='issue'                                    # vgw-<random hex>, defaults
make key ARGS='issue --key-id vgw-acme-01 --tenant acme --user alice'
make key ARGS='issue --key-id vgw-acme-01 --pool kimi'   # attach an upstream pool
make key ARGS='issue --key-id vgw-acme-01 --upstream-key sk-...'
make key ARGS='issue --key-id vgw-acme-01 --rate-limit-rpm 100 --token-budget 5000000'
make key ARGS='list'                                     # KEY_ID / TENANT / USER / ACTIVE / CREATED
make key ARGS='show vgw-acme-01'                         # record plus its upstream mapping
make key ARGS='map vgw-acme-01 --pool kimi'              # set or change the mapping
make key ARGS='map vgw-acme-01 --upstream-key sk-...'
make key ARGS='map vgw-acme-01 --none'                   # clear (use gateway OPENCODE_API_KEY)
make key ARGS='revoke vgw-acme-01'                       # active=false, record kept
make key ARGS='pool list'
make key ARGS='pool create kimi'
make key ARGS='pool add kimi k1 sk-...'
make key ARGS='pool remove kimi k1'
make key ARGS='pool disable kimi k1'
make key ARGS='pool enable kimi k1'
make key ARGS='pool reset kimi'

Run make key ARGS='help' for the full flag list. The scripts call the OpenBao KV v2 HTTP API directly (not APISIX or etcd) through podman exec gw-openbao, because OpenBao publishes no host port. Writes carry the shared OPENBAO_TOKEN, so they are not attributed to an individual operator.

Configuration

Routes and config (control plane)

flowchart TB
    J2[apisix.yaml.j2]
    YAML[apisix.yaml]
    Seed[seed-routes.sh]
    Etcd[(etcd)]
    Routes[APISIX routes]
    Admin["Admin API :9180"]

    J2 --> YAML --> Seed --> Etcd --> Routes
    Admin -.->|route CRUD| Etcd

Traditional/etcd mode (RBAC-enabled; APISIX authenticates as a dedicated etcd user): routes live in etcd, seeded from the rendered conf/apisix.yaml on stack start. Admin API and built-in dashboard are reached via podman exec gw-apisix curl http://127.0.0.1:9180/ui/ (no host port).

Key Files

• conf/config.yaml: APISIX traditional/etcd mode: plugin list, shared dicts, env vars, Admin API, Prometheus port
• conf/apisix.yaml: Committed route render (18 routes); drift-checked against conf/apisix.yaml.j2
• conf/apisix.yaml.j2: Jinja2 route template rendered at deploy from .env
• res/scripts/seed-routes.sh: Seeds etcd from rendered apisix.yaml on stack start
• conf/openbao.hcl: OpenBao production config (file-storage backend)
• conf/prometheus.yml: Prometheus scrape config (APISIX :9100)
• conf/profanity/: vendored rejection-language dictionaries (refresh: make gw-update-dictionaries)
• conf/grafana/: Grafana datasources + 5 provisioned dashboards
• conf/redact-patterns.json: PII detection: 6 regex patterns + 2 dictionary categories
• conf/sql/: All SQL (no inline SQL anywhere). clickhouse-init.sql base schema, migrations/ incremental changes, plus ops/, ingest/, grafana/queries/, sqlite/, tests/. Templates rendered by res/scripts/lib-sql.sh and linted by sqlfluff via .sqlfluff
• conf/vector.toml: Vector pipeline: HTTP source, VRL remap (parse_json for model extraction), ClickHouse sink
• res/docker/docker-compose.yml: 8 services: apisix, etcd, clickhouse, migrate, vector, openbao, prometheus, grafana
• res/docker/Dockerfile.apisix: Custom APISIX image: Lua plugins + config copied in
• res/docker/Dockerfile.openbao: Custom OpenBao image (production file-storage)
• res/docker/openbao-entrypoint.sh: OpenBao auto-init, auto-unseal, gateway key provisioning (data persists via openbao-data named volume)
• .env: Secrets: ADMIN_KEY, OPENCODE_API_KEY, GATEWAY_API_KEY, OPENBAO_TOKEN

Environment Variables

ClickHouse Tables

Telemetry runs in response/log phases after the upstream returns:

flowchart TB
    Response[Upstream response]
    Tele[Telemetry plugins]
    CH[(ClickHouse)]
    Vector[Vector]

    Response --> Tele
    Tele -->|usage_log| CH
    Tele -->|request_log| Vector
    Vector --> CH

sse-usage writes usage_log directly (as apisix_rw); http-logger ships full request/response metadata to Vector, which inserts request_log (metadata) and request_bodies (bodies; req_body is PII-tokenized by the redact plugin before logging, and is invisible to the Grafana datasource).

Retention moves old parts to a tiered, ZSTD-recompressed archive volume rather than deleting them. The ops-health storage panel monitors growth, and nightly backups land on /mnt/ws-backup.

Grafana Dashboards

flowchart TB
    Plugin[prometheus plugin]
    Export["APISIX :9100"]
    Prom[(Prometheus)]
    Grafana[Grafana]
    CH[(ClickHouse)]

    Plugin --> Export
    Prom -.->|scrape| Export
    Grafana -.->|PromQL| Prom
    Grafana -.->|SQL queries| CH

The prometheus plugin exports request metrics at :9100. Prometheus scrapes every 15s (conf/prometheus.yml). Grafana uses Prometheus for ops panels (latency, error rate) and ClickHouse for cost and usage. Grafana only queries data; it does not write.

Five provisioned dashboards (default: now-90d lookback, 5s refresh):

Cost & Usage breaks spend down by token category and model:

Operations & Health covers live traffic, error rate, and status codes:

Model Experience ranks each model by a behavioural usefulness score built from satisfaction, friction, and abandonment signals:

Model Performance tracks speed, stream reliability, and time to first token:

The leaderboard shows top clients (p20) and top models (p21) by cost and tokens. After editing dashboard JSON, run make gw-restart-grafana to reload provisioning.

opencode Integration

The gateway registers canonical auth-mode providers: OpenCode Go virtual/API key, OpenCode Zen API key, llamafile no-auth, three Moonshot Kimi modes (device OAuth, virtual key, and API key), and Z.ai GLM API-key passthrough as custom providers in opencode.

# Refresh the gateway-side provider/model catalog (does NOT touch client config)
make sync-models

# Install ALL gateway providers into opencode config (auth skipped by default)
make setup-providers

# Same, prompting for API keys / running OAuth device flows:
make setup-providers REQUIRE_AUTH=1

# Add a key for one provider later:
bash res/scripts/opencode-provider-login.sh --provider-id workspace-gw-zai-api-key --require-auth

This enriches the gateway catalog from /opencode_federated/v1/models (Go tier, *-free models filtered out), /opencode_zen/v1/models (all Zen/free models), and /llamafile/v1/models (local llamafile upstream), with canonical metadata (name, context limit, capabilities, cost, modalities) from models.dev, including variants (reasoning-effort presets) derived from models.dev reasoning options exactly as opencode derives them. Provider entries are written into ~/.config/opencode/opencode.jsonc by the login script above, which fetches each ready-made block from /gateway/providers/<id>/opencode:

• workspace-gw-opencode-go-virtual-key: virtual-key mode, Go tier
• workspace-gw-opencode-go-api-key: API-key passthrough, Go tier
• workspace-gw-opencode-zen-api-key: API-key passthrough, free/Zen models
• workspace-gw-llamafile-no-auth: no-auth local LLM
• workspace-gw-kimi-device-oauth: device OAuth (gateway-managed Kimi token)
• workspace-gw-kimi-virtual-key: virtual-key mode for Kimi (vgw-*)
• workspace-gw-kimi-api-key: API-key passthrough for Kimi
• workspace-gw-zai-api-key: API-key passthrough for Z.ai GLM (Coding Plan endpoint)
• workspace-gw-anthropic-passthrough: API-key passthrough for Anthropic
• workspace-gw-anthropic-device-oauth: device OAuth for Anthropic
• workspace-gw-openai-device-oauth: device OAuth for OpenAI (ChatGPT)
• workspace-gw-alibaba-token-plan-passthrough: API-key passthrough, Alibaba Token Plan
• workspace-gw-alibaba-token-plan-cn-passthrough: API-key passthrough, Alibaba Token Plan (China)

For the OAuth providers, the login script starts the device flow and prints the verification URL.

Each provider receives the full enriched catalog, because opencode drops providers that expose zero models. The two Go providers (workspace-gw-opencode-go-virtual-key and workspace-gw-opencode-go-api-key) filter out *-free models, since their relay rewrites to /zen/go/ (paid only); free models are served only through workspace-gw-opencode-zen-api-key. The llamafile provider takes its model list from /llamafile/v1/models (or a default id when the server is down); MiniCPM5 uses context 131072 (scaled to 104857 at 80%) with tool_call: true. The script runs automatically on make gw-start and make gw-restart via the Ansible playbook.

Context limits are scaled by CONTEXT_LIMIT_PCT (default 80) from .env, so e.g. CONTEXT_LIMIT_PCT=80 reduces a 200000-token context to 160000. An absolute ceiling CONTEXT_LIMIT_CEILING (default 128000) is then applied: any scaled value exceeding the ceiling is clamped to it. Set to 0 to disable.

Result in opencode config:

{
  "provider": {
    "workspace-gw-opencode-go-virtual-key": {
      "api": "http://localhost:9080/opencode_federated/v1",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:9080/opencode_federated/v1",
        "apiKey": "vgw-gateway-key",
        "headers": { "X-Tenant-ID": "default", "X-User-ID": "agent" }
      },
      "models": {
        "minimax-m3": {
          "name": "MiniMax M3",
          "family": "minimax",
          "release_date": "2026-06-01",
          "attachment": true,
          "reasoning": true,
          "temperature": true,
          "tool_call": true,
          "cost": { "input": 15, "output": 75, "cache_read": 1.5, "cache_write": 18.75 },
          "limit": { "context": 160000, "output": 24000 },
          "modalities": { "input": ["text", "image", "pdf"], "output": ["text"] },
          "status": "active"
        }
      }
    },
    "workspace-gw-opencode-go-api-key": {
      "api": "http://localhost:9080/opencode/v1",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:9080/opencode/v1",
        "headers": { "X-Tenant-ID": "default", "X-User-ID": "agent" }
      },
      "models": { "...": "same enriched models as workspace-gw-opencode-go-virtual-key" }
    }
  }
}

Testing

make test          # Run all stages (excludes live upstream API tests)
make test-live     # Run all stages including live upstream API tests
make gw-test       # Same as test, against the running stack

1. Lua unit tests via resty CLI inside the APISIX container
2. Config validation: 25 scripts (YAML, SQL, TOML, JSON, dashboard structure, migrations)
3. Reconciler static analysis: syntax, strict mode, error handling
4. Integration: black-box HTTP against the running stack (llamafile e2e, event_id alignment, data flow, cost e2e, Grafana panel checks)
5. Repository hooks: pre-commit and pre-push hooks present and wired
6. E2E: real Go API calls (gated behind RUN_LIVE_API_TESTS=1)

Make Targets

Gateway Lifecycle (systemd-managed)

The stack is owned by the systemd user unit gateway-compose (make gw-deploy installs it with linger). Start/stop/restart go through systemctl so an unmanaged compose stack never fights the unit's Restart=always.

Notes: gw-restart force-recreates every container from the current compose and does not preserve running containers or apply compose network changes selectively. gw-recreate-service applies network and env changes and supports grafana, clickhouse, vector, openbao, prometheus, and etcd.

Key Management

Quality Gates

License

• Apache APISIX 3.18.0: Apache 2.0
• OpenBao 2.4.4: MPL 2.0
• ClickHouse 24.8: Apache 2.0
• Vector 0.40: MPL 2.0
• Prometheus v3.13.1: Apache 2.0
• Grafana 13.0.2: AGPLv3