Millwright Review 2026: A Self-Hosted LLM Router in One Rust Binary

When every request hits your most expensive frontier model, the bill climbs fast — and provider switches quietly break prompt-cache reuse. Millwright is a self-hosted LLM router (a single Rust binary) that puts the cheaper path in charge, with policy, cache affinity, and spend control in one place.

What is Millwright?

Millwright is an open-source LLM router that sits between your AI apps and the model providers you choose. It accepts OpenAI Chat Completions and Anthropic Messages, then routes each request to an OpenAI-compatible API, Anthropic, or Amazon Bedrock. You define which models fill the cheap, mid, and frontier roles; Millwright picks the lowest estimated-cost healthy route your policy permits.

It’s a router, not an agent orchestrator — it doesn’t spawn agents, inspect prompts, or rewrite context.

Key features

• Policy-controlled routing — classify requests into cheap / mid / frontier roles; high-risk and planning work always uses frontier.
• Cache-aware affinity — keeps independent cheap/mid/frontier lanes under one session ID to protect prompt-cache reuse.
• Spend ledger — SQLite by default, PostgreSQL for production, with routing provenance.
• Provider portability — change providers behind one endpoint without touching app code.
• Runs the data plane yourself — provider credentials stay on your infrastructure.
• Works with Claude Code, Codex, OpenCode, Pi, and other compatible clients.

Who should use it?

Millwright is for teams running many concurrent agent sessions who want to cap LLM spend without hand-tuning every call. If you’re already comparing routers, it’s a self-hosted alternative to LiteLLM and hosted gateways — no required control plane.

It’s less useful if you only call one model from one app; the setup cost outweighs the benefit there.

Pros and cons

Pros: self-hosted and private, explicit spend control, cache-aware concurrency, inspectable routing.

Cons: early release (v0.1.0) so the API may shift; router-only (no agent orchestration); needs Rust 1.97+ to build from source.

Pricing

Free and open source (Apache-2.0).

FAQ

Does Millwright inspect my prompts? No. Routing decisions use task/risk headers and model selection, never message content.

Can it replace LiteLLM? It covers core routing, caching, and spend-tracking use cases and is fully self-hosted; whether it replaces LiteLLM depends on which advanced features you rely on.

Which clients work with it? Any OpenAI-compatible or Anthropic-compatible client, including Claude Code, Codex, and OpenCode.