LangChain: Model Routing in the Harness Cut Coding Agent Costs 64%

Why routing instead of one frontier model

LangChain starts from a cost problem: frontier models remain expensive, and that becomes prohibitive as agents run at massive scale. The company says its own monthly coding agent spend began to climb rapidly, and that customers voiced the same concern.

Its argument is that most agents do not need frontier-level intelligence for every task. It cites Anthropic's guide to choosing a model, which notes that "for many applications, starting with a faster, more cost-effective model like Claude Haiku 4.5 can be the optimal approach." Past a certain point, LangChain says, a more capable model adds little quality while cost and latency keep climbing.

LangChain describes a good agent as having model-harness-task fit: the right model with the right context for a given task. A router picks that model for each task. According to the company, the decision should live in the harness because it already assembles the domain and task context that a gateway typically lacks.

Mapping tasks and picking three models

The test bed was Open SWE, which LangChain engineers use from Slack and a web UI to ask questions about codebases and request changes. The team pulled thread-level data from LangSmith traces, including request types, cost and turn count per thread, and used an LLM classifier to label a week of interactive threads by task type.

Code changes dominated. New features made up 22% of threads and bug fixes 17%, followed by test or no-op runs at 16%. Feature work tended to produce longer, costlier threads with higher median turn counts, while testing and release threads were relatively short and inexpensive. At the time, every Open SWE thread went through a top-tier frontier model. That spread of complexity led to the hypothesis that a router could infer task type and difficulty from the first request and send it to a cheaper or faster model.

For the models, LangChain used the Artificial Analysis Intelligence Index, which scores models on common tasks and reports cost per task. It chose three points along the curve, using data as of Sep 9, 2026:

• Fast: GLM-5.3-Flash (xhigh)
• Balanced: GPT-5.6 Sol (medium)
• Performance: GPT-6 Astra (low)

The models come from different providers, and the fast tier is an open model. LangChain says GLM-5.3-Flash sits on the Pareto frontier next to closed models. Because LangChain offers a generic model interface across providers, it says swapping in a better model is a one-line change to the router.

How the router works

The routing logic sits in LangChain middleware, which can swap the model an agent calls without changing anything else about the agent. In Open SWE the router runs on the thread's first human message and has three parts: a base prompt telling the classifier to pick the least expensive model likely to complete the task, short plain-language criteria for each tier, and a classifier model that reads the request and picks a tier.

LangChain wrote the criteria from its own task analysis and from provider guides for GPT-5.6 Sol, GPT-6 Astra and GLM-5.3-Flash. Because the criteria are written for Open SWE's task set, the router is deeply coupled to those tasks, which the company offers as the reason it belongs in the harness.

The first version used an LLM with structured output. The classifier now runs on Jev, a newly released decision model, which according to LangChain made classification almost 50× faster. The router picks a model once at the start of each thread and keeps it for the whole thread. LangChain acknowledges that this naive design does not handle a thread that changes topic or complexity mid-flight, and says it addresses that in a later section on what is next.

Measuring whether quality holds

LangChain stresses that lower cost only counts if quality does not regress. The chosen model must be able to finish the task and should be the cheapest, fastest one that can. It describes two ways to track this. Offline evals run the router against a fixed dataset, which is safe and repeatable but needs data that resembles real traffic and is graded on what users care about. For a coding agent that means PR quality and reviewability, which the company says are hard to grade offline. A/B tests split live threads between the router and a single-model baseline, giving representative data at the risk of exposing users to a potentially non-optimal router.

LangChain ran A/B tests with two signals. Merged PRs per thread became the main success metric, since Open SWE records whether each PR it opens is merged or closed. The team also added thumbs up and down ratings, logged as feedback on the LangSmith trace. Merged PRs were the stronger signal, the company says. Feedback was sparse, but it surfaced routing mistakes.

What it means for agent builders

The result is one team's experiment on one agent, and LangChain notes the router is tuned to Open SWE's workload. Still, the approach it lays out is general: analyze your own task mix, choose models along the cost and intelligence curve, write tier criteria from that analysis, and measure outcomes with evals or live tests. LangChain's own conclusion is that effective router design is rooted in observability and evals, and requires an understanding of an agent's task space, model performance research and clear success criteria.