Senior DevOps Engineer
DevOps Engineer
Intelligent model routing promises a smaller bill with no quality loss. We decided to see for ourselves.
Not every request needs the most capable model you have, but asking engineers to pick per request is a non-starter. The proposed solution is Intelligent routing, specifically semantic routing. A router sits in front of the models and sends each request to the model best suited for completing it.
The vLLM Semantic Router (vSR) claims to perform that type of intelligent routing. A research paper from the team behind vSR reported dramatic reductions in both token usage and latency, with no accuracy loss by routing between a single model with reasoning on vs off. These claims were compelling enough that we decided to put them to the test ourselves for our model choice use case.
TL;DR; after tuning¹, using the vLLM Semantic Router to dynamically route between two models with different capability and cost profiles reduced cost by roughly 33%, while achieving a pass rate of 43% compared to 48% when exclusively using the higher-capability model.
What vLLM Semantic Router does, and what the paper did not test
vLLM Semantic Router sits in front of your models and decides which model should receive each request. Our router configuration scores each request on keyword, embedding² and complexity signals, combines them into one number, and compares that to a threshold². Above the line, the request goes to the strong model. Below it, to the cheaper one. Three things shape those decisions:
The example requests (vSR calls them exemplars) and keywords the router matches against
How much weight each signal gets
And where the threshold sits
Those are the knobs we tuned. vSR ships recipes and a reference config, not a plug-and-play two-model default. Operators are expected to supply their own example requests, signal weights, and threshold. We wrote our first round config from scratch; three rounds of tuning refined it.
The paper behind vSR's efficiency claims is When to Reason: Semantic Router for vLLM. That paper did not route between two models, like we did. Instead, the paper’s experiment served a single model, with the router selectively enabling reasoning based on the complexity of each request. On MMLU-Pro, an exam benchmark, it scored 10.24 percentage points above direct inference with the same model, with 47.1% lower latency and 48.5% fewer tokens. Those numbers describe a router deciding how much one model should think. They are not a forecast for routing between two different models, and we did not try to reproduce them.
We kept the paper's router and changed the question. We tested vSR on 62 agentic tasks from Terminal-Bench 4.0, a recently released and comparatively difficult benchmark of terminal-based work across a variety of subjects, and ran each task three times. The paper varied how much one model thinks. Instead of using thinking as the variable, we varied which model requests were sent to. The router chose between Claude Sonnet 5.5 at medium effort³ and Claude Opus 5.5 at high effort. We fixed effort on purpose, so the two models kept a clear performance gap for the router to choose between. In our setup, vSR was attached to the Linux Foundation's AgentGateway as an external processor, and the gateway logged the model requested, the model served, tokens, and cost for every call.
The result: a third cheaper than Opus, but solving nine fewer tasks
After tuning, using the vLLM Semantic Router reduced cost by about a third against always using Opus and solved 43% of the benchmark tasks against Opus's 48%. Costs are from the gateway's request logs, which matched Anthropic's bill within about 1% for the routed runs.
186 = 62 tasks, each run three times.
One asterisk on the headline number: these results are on the same 62 tasks we tuned against. We did not measure how the router performs on tasks it hasn't been fitted to. See Objections for the full version.
Routing also made requests slower. The paper's 47.1% latency cut came from skipping reasoning, not from choosing between models. At the median, a routed request waited 7.4 to 9.6 seconds before its response started, where always-Opus was 2.2 seconds. Before choosing a model, the router turns each request into an embedding so it can compare it with its example requests. We ran that process on the same small cloud servers as the gateway, and it slowed further when requests arrived at the same time. vSR's documentation describes faster settings that trade some precision for speed. We used the slowest, most precise setting.
How we tuned it
Tuning took three rounds. We sent each of the 62 TB4.0 task instructions to the router as a bare API call, with no agent or harness, and recorded which model it picked. We compared those decisions to the results of our Sonnet-only and Opus-only (Claude Code orchestrated by Harbor), where we knew which tasks each model had failed or succeeded. We then reworked the examples the router matches against and adjusted how it weighs each request.
We guarded against one kind of contamination: no example request copies or closely paraphrases a TB4.0 task. We checked this two ways, by manually auditing every example and by passing them through the same MMBERT-Matryoshka embedding model vSR uses to confirm they were sufficiently distinct from the task prompts. The threshold and weights are a different story; see Objections.
What any router can and can't reach
Two arms bound what any two-model router can do on this benchmark. Always using Sonnet is the floor for a router that sends too much to the cheaper model: 57 tasks for $191. Always using Opus is the ceiling for one that sends too much to the stronger model: 89 tasks for $554. Ours landed between the two: 80 tasks for $369. A perfect "oracle" router that sent each task to whichever model solves it would reach 93 tasks, slightly above Always-Opus because several tasks were consistently solved by Sonnet but not by Opus in our runs.
Those bounds have two consequences for routing in general. First, any win comes from cost, not accuracy: the oracle is only four tasks above Always-Opus. Second, the tuned router costs 1.9 times as much as Sonnet alone, so if your work is Sonnet-shaped, routing is a bad trade.
Noise is the other reason to be careful about any single number here. Between identical repeats, 26% to 37% of tasks flipped between pass and fail, and single-run pass rates spread from 22.6% to 56.5% across our runs. We ran every arm three times for that reason. Nine solved tasks between the tuned router and Opus is a real gap, but a noisy one.
Objections
"You tuned on the test." Partly, and we would rather say so. There are two kinds of contamination, and we handled them differently. The first is input contamination: whether TB4.0 task text ended up in the router's example requests. We checked for this, as described above, and it did not. The second is tuning-signal contamination: the threshold, signal weights, and example request choices were adjusted based on how these 62 tasks scored. We did. The router doesn't contain the task text, but its knobs are fitted to the task outcomes, so a held-out set of agentic tasks (ones the router had never been scored against) would almost certainly show lower numbers. In real use there is no test set⁴ to protect. You would tune on your own team's actual work. Read our savings as a result on work the router was tuned against.
"You didn't tune it enough." We tuned over three rounds, and each round the router sent more of the 62 task prompts to the right model. It still gave up some accuracy. We do not claim a fourth round would not do better.
"This only works for varied work." Agreed. Terminal-Bench stands in for varied workloads, where a router has real choices to make. We did not test workloads with consistent requests, where you would pick the one best-fit model and likely skip routing.
"You left out session protection⁵." We ran stateless per-request routing as a baseline for cost and accuracy. vLLM Semantic Router's session protection holds a session on one model. We did not enable it, so we cannot say whether it recovers savings.
What to do with this
Already run a gateway, with varied work? Try a router, and plan to tune it. Ours saved about a third against Opus after three rounds of tuning, and still gave up some accuracy.
Tune on your own team's real tasks, not ours. In a real deployment there is no test set to protect, so use the exact work your engineers do. Our results come from tuning against benchmark tasks. Yours will come from yours.
Consistent requests? Pick the model that fits them. A router has little to choose between there, and we did not test it.
Decide what a solved task is worth. Between the tuned router and Opus, the nine extra tasks Opus solved cost about $185 more, roughly $20 each. If a task is worth more than that to you, pay for Opus.
Sort tasks into always solved, never solved, and mixed. A router can only earn its keep on the mixed ones.
Where this leaves us
This is a result for one router implementation on one kind of work, not for routing as a concept. Tuned, vLLM Semantic Router cut our Opus bill by about a third for nine fewer solved tasks. Decision endpoints⁶ are the next option we would look at, but we have no follow-up planned while the options are limited.
If this problem sounds familiar, reach out to Liatrio. We would love to help you solve it.
Learn more about how we help teams scale AI.
Glossary
Tuning. Adjusting the example requests the router matches against, how much weight each signal gets, and where the threshold sits.
Embedding, signals, threshold. An embedding is a numeric fingerprint of what a request means, so requests can be compared. Signals are the scores the router computes from a request. The threshold is the cut line between sending a request to the cheaper model and the stronger one.
Effort. A setting for how much a model thinks before it answers. Higher effort means more thinking per answer.
Test set. Tasks kept out of the tuning process, so a result can be trusted on work the router has not seen.
Session protection. A vLLM Semantic Router feature that keeps a session on the model it started on and moves it up only for a hard request. We did not enable it.
Decision endpoint. An endpoint that fronts a model used to make decisions, such as which model should answer, rather than a large language model that generates text.
