KTMS is a self-hosted LLM gateway: it keeps your vendor keys, routes each request by effort, falls back when a provider fails and caps spend. light-tokenmeter shows who burns the tokens. ktms-admin puts it all in your browser.
KTMS, measured in October 2026 on GCP c3-standard-4 against a mock upstream. Compiled proxies such as agentgateway, APISIX and Kong add even less latency; KTMS plays in their league and adds the features. Method and full numbers →
Use them together or one at a time. Each is a single Go binary with its own Docker image, Helm-friendly defaults and an MIT licence.
Your services talk to one endpoint. KTMS holds the vendor keys, picks the model, retries elsewhere when a vendor is down and records every call.
ai-processJudgeYour apps, or KTMS, push one event per model call. Ask who is spending your tokens, and whether they are worth it.
COPYManage models, routing rules and keys, try a prompt in the playground and read the spend dashboard. No more grpcurl.
Clients send effort: LOW | MEDIUM | HIGH and free-form attributes. An ordered rule list maps them to a chain of models with fallbacks.
Rules change at runtime over the admin API or in ktms-admin. A new model release is a config change, not a refactor across a dozen services.
Run Ollama, vLLM or llama.cpp on your own hardware. ktms-node-agent registers each machine and reports VRAM, time to first token and tokens per second.
KTMS balances the pool with Smooth WRR, keeps hot or overloaded nodes out of rotation and, with nodes_first, spills over to cloud vendors only when local capacity runs out.
Per-model requests-per-minute and daily token budgets for every API key and every end user, enforced in Redis before the vendor ever sees the request.
Non-interactive work goes through the vendors' Batch APIs at a 50% discount, with background polling and a synchronous fallback.
KTMS sends one usage event per call to light-tokenmeter. Slice spend by project, user, action, vendor or model, with failure rates and latency next to the cost.
ktms-admin turns it into a verdict: the workloads that burn tokens for little value, and the ones that earn them.
Your services keep a single address and a scoped ktms_… key. Everything behind it can change without a deploy.
git clone to a routed requestDocker Compose brings up KTMS with Postgres and Redis, seeds default models and rules and a dev key.
Change base_url and use model: "auto". Routing, fallbacks and limits come for free.
Docker images for amd64 and arm64, a Helm chart and a production checklist are in the KTMS docs.
git clone https://github.com/ai-process/ktms.git cd ktms # any of OPENAI_API_KEY, GEMINI_API_KEY, ANTHROPIC_API_KEY OPENAI_API_KEY=sk-... docker compose up -d --build curl -s http://localhost:8080/readyz # {"status":"ready"}
curl -s http://localhost:8080/v1/chat/completions \ -H "Authorization: Bearer ktms_dev_default_key" \ -H "Content-Type: application/json" \ -d '{"model":"auto","messages":[{"role":"user","content":"Say hi in three words."}]}' # response headers tell you who answered and why: # X-KTMS-Vendor, X-KTMS-Rule, X-KTMS-Placement
from openai import OpenAI client = OpenAI(base_url="http://localhost:8080/v1", api_key="ktms_dev_default_key") response = client.chat.completions.create( model="auto", # or "auto:low", "auto:high" messages=[{"role": "user", "content": "Say hi in three words."}], ) print(response.choices[0].message.content)
No enterprise edition, no paywalled security features, no licence keys and no telemetry. Fork it and run it forever.
Static Go binaries and small images that start in milliseconds, fit scale-to-zero and leave the memory to your models.
Protobuf APIs with streaming and reflection for services, plus an OpenAI-compatible HTTP API for everything else.
Guides for every service, from a single Docker command to a GPU pool in production.
How the services fit together, with one Compose file for the whole stack.
APIs, effort routing, fallbacks, limits, batches, speech and images.
Self-registering nodes, health states, Smooth WRR and cloud fallback.
Connect Ollama, vLLM, TGI or llama.cpp to KTMS step by step.
Events, batches, summaries and the greedy vs worthy analysis.
Sign-in, allowlists, gateway management and the spend dashboard.
Architecture patterns, benchmarks with the method attached, and what we learn while building KTMS.
Free, self-hosted and MIT-licensed. Run it next to your current setup and compare.