Measured, not guessed
Cut your inference bill
by up to 97%.
adaptroute puts a small model in front of every request, routes only
the hard ones to a big model, and — on the edge — distills the big model into the
small one so the bulk runs free. A promotion gate and a live watchdog mean it gets
cheaper over time without ever getting worse.
MONTHLY COST · 1M requests · ~1,200 in / 200 out tokens
illustrative, at published rates
Everything → Sonnetthe lazy-safe baseline
Haiku + Sonnet, routedAPI only · no infra
Local 4B + teacherweight-distilled · edge
Two ways to run it
Yes — Haiku for the small model,
Sonnet for the big one. Here's the trade.
The cheap model is whatever you point adaptroute at. You can use Claude
Haiku as the small tier and Sonnet as
the big one and start saving with zero setup — you just give up the single biggest lever,
weight distillation, which needs a model you can train.
Mode A · API only
Haiku (cheap) + Sonnet (costly)
Fastest to try · nothing to host
~42% cheaper
- Judge routing — each query-type goes to Haiku or Sonnet by measured accuracy
- Adaptive tool-pruning — Haiku sees only the tools its query-type needs
- Context distillation — Sonnet's examples ride in Haiku's prompt
- No weight distillation — you can't train a closed model, so Haiku never bakes the skill in, and it isn't free per token
ADAPTROUTE_STUDENT_URL = api.anthropic.com
ADAPTROUTE_STUDENT = claude-haiku-4-5
ADAPTROUTE_TEACHER = claude-sonnet-5
Mode B · Edge · recommended
Local 4B (cheap) + any teacher
Runs your own small model · maximum savings
~85–97% cheaper
- All three distillations — context, reasoning-trace, and weight (LoRA)
- The bulk runs at ~$0 — a local Qwen/Phi/Gemma handles most traffic
- Gets permanently smarter — retrains nightly on your traffic; the gate blocks any regression
- Teacher is anything — a big local model, or Sonnet/GPT/DeepSeek via API
ADAPTROUTE_STUDENT = Qwen/Qwen3.5-4B
ADAPTROUTE_STUDENT_OLLAMA = qwen3:4b
ADAPTROUTE_TEACHER = claude-sonnet-5
Same product, one env var apart. Start on Mode A
to prove the savings with no infra; move the cheap tier to a local model
when you want it free and self-improving.
The math
Where the money goes
1M requests/month at ~1,200 input / 200 output tokens, published per-token rates.
Real numbers depend on your traffic mix — adaptroute measures yours and routes on the evidence.
| Setup | Small model | Big model | $ / month | vs baseline |
|---|
| Everything to the big model | — | Sonnet | $6,600 | baseline |
| Routed, API only | Haiku | Sonnet | $3,830 | −42% |
| Edge, weight-distilled | Local 4B | any teacher | $1,175 | −82% |
| Edge, fully distilled | Local 4B | 3% judge only | $275 | −96% |
How it works
Three kinds of learning, one router, one guarantee
Make it smart
Three kinds of distillation
Context (examples in the prompt) → reasoning traces (the teacher's chain-of-thought) →
weight (a LoRA baked into the small model). Each cheaper to run than the last; the last is free forever.
Make it fit
Learned tool selection
The big model sees the whole tool catalog; the small model sees only the few tools its
query-type actually uses — measured live, ~89% fewer schemas, so a small model fits a big catalog.
Make it cheap
An LLM judge routes
On a small sample a costly judge grades cheap-vs-costly head-to-head and sends each
query-type to the cheapest tier that's proven good enough. The rest is a config lookup — no judge in the hot path.
Keep it safe
Gate + watchdog = never regress
A promotion gate ships an update only if it's measurably better and hasn't forgotten anything;
a live watchdog auto-rolls-back if quality dips. Safe to run unattended.
Infrastructure
From a Raspberry Pi to a GPU — you pick
The routing brain is tiny — an 88 KB install, pure Python. The only heavy
part, fine-tuning the small model, is periodic and separable. So you can serve almost anywhere and
rent muscle only when you retrain.
No hardware
Mode A · API only
Both models are API calls; adaptroute is just the router. Runs on a laptop, a container, or a Pi.
On the edge
Mode B · a box you own
Run the small model locally — a Raspberry Pi 5 (quantized 1–4B, low volume), a Mac mini or Jetson
(more throughput), or a GPU box (fast, high volume).
RECOMMENDED FOR A PI · serve on the edge, fine-tune in the cloud
a few cents / night
Raspberry Pi 5
Serves & routes
always on · private · ~$0
- Answers the bulk on a small local model
- Prunes tools, logs traces, runs the watchdog
- Just pip install adaptroute — no GPU, no torch
Cloud GPU sandbox
Fine-tunes & gates
spot GPU · minutes a night
- Wakes nightly, pulls the day's traces
- Runs the LoRA fine-tune + promotion gate
- Publishes a versioned adapter — the Pi promotes it, the watchdog rolls it back on a dip
Design-partner program
Become a founding design partner
adaptroute is in a hands-on private beta — we set it up
with you and support it directly, and your code stays yours.
$1,000 / month — a founding design-partner rate, locked through the beta —
with full white-glove onboarding & support; cloud/GPU billed at cost (a few $/mo) or bring your own. Email
gopalkris@adaptroute.dev.
Once you're set up, the whole stack is one install and one command, configured with
ADAPTROUTE_* env vars:
# install (add [train] for the edge / weight-distillation mode)
pip install adaptroute[train]
# wire up: registry + nightly train-and-deploy + watchdog cron
adaptroute install
adaptroute status # show config + installed jobs
# watch it — served accuracy, % routed cheap, spend saved, tools pruned
adaptroute dashboard # → http://localhost:8080
# prove the safety loop
adaptroute watchdog --simulate # healthy → dip → auto-rollback
Point it at your cheap and costly models
| What | Env var | Example |
|---|
| Cheap model — the one that answers most requests | ADAPTROUTE_STUDENT | Qwen/Qwen3.5-4B · or claude-haiku-4-5 |
| …its served name (edge mode) | ADAPTROUTE_STUDENT_OLLAMA | qwen3:4b |
| Costly model — teacher + escalation target | ADAPTROUTE_TEACHER | claude-sonnet-5 · or qwen3:32b |
| …its endpoint | ADAPTROUTE_TEACHER_URL | any OpenAI-compatible URL |
| Tool catalog — big model sees all, small sees the learned subset | ADAPTROUTE_TOOLS | calculator,web_search,… |
| Alerts on every promote / rollback | ADAPTROUTE_ALERT_WEBHOOK | your Slack/Teams webhook |
Contact
Talk to the founders
Notes & disclaimers
-
·
Cost figures are illustrative, at published per-token rates for a sample
workload. Actual savings depend on your traffic mix and are not guaranteed.
-
·
Weight distillation — the self-improving lever — requires an open model you run
and can fine-tune. Closed API models (Claude, GPT) support routing, tool-pruning, and in-context
distillation, but cannot be weight-distilled.
-
·
Measured accuracy figures are on specific tool-choice tasks and small samples — indicative, not a
performance guarantee.
-
·
adaptroute is in private beta and provided as-is. Cloud/GPU is billed at cost or brought by you.
-
·
Pricing shown is a founding design-partner rate, locked through the beta;
general-availability pricing will differ.