Fivetran study finds coding model price is a poor value guide

A blinded test in everyday coding work

Fivetran conducted the study from September 14, 25, 2026, with engineers voluntarily assigned two of the three models, one per week and in random order. The models were presented through aliases, while a LiteLLM proxy removed model and provider names from responses and instructed the models not to reveal their identities.

Of the 39 participating engineers, 35 used at least one assigned model and 31 tested both. The models ran through Claude Code, the participants’ usual coding agent, and the company recorded model preference, spending, and pull request activity in normal repository work.

Fivetran said the design was intended to avoid results being shaped by brand familiarity or expectations about price. Unlike public benchmarks focused on isolated tasks, the test examined how the models performed in the company’s own repositories and tools.

Sol led preferences, but the result was not decisive

Among the 27 engineers who completed the final survey after using both assigned models, Sol was preferred in both of its matchups. Six of nine engineers preferred Sol to Luna, while seven of 10 preferred Sol to Terra. Terra edged Luna by five votes to three.

The differences were narrow. Fivetran reported that 12 of the 27 preferences were weak, including four of Sol’s seven wins over Terra. The company said none of the three splits was clearly more than chance, and all exact 95% confidence intervals crossed 50%.

Pull request activity also did not establish a productivity ranking. A typical engineer opened five pull requests during the Luna week, four during the Sol week, and two or three during the Terra week. Fivetran said those counts largely reflected whether engineers worked on a large refactor or several smaller changes.

Higher prices brought sharply higher spending

The token prices varied widely. Terra’s input rate was 10 times Luna’s, while Sol’s was 20 times Luna’s. The listed unchached input prices were $0.20 per million tokens for Luna, $2.00 for Terra, and $4.00 for Sol. Output prices were $1.20, $12.00, and $20.00 respectively.

Typical weekly spending followed the same pattern. The median engineer spent $5.32 during the Luna week, $41.42 during the Terra week, and $168.44 during the Sol week. Average spending was higher, at $9.95 for Luna, $88.85 for Terra, and $243.49 for Sol, because some engineers used the models much more heavily.

Fivetran concluded that price tier was a poor proxy for day-to-day value. According to the company, the 20-fold token price increase from Luna to Sol delivered a modest preference edge rather than a decisive win.

Task fit affected the user experience

Fivetran said heavier models could be useful for large refactors, where thorough planning paid off. For focused changes, however, engineers placed more value on fast responses and tight scope control, and heavier models sometimes produced extra steps, broader edits, or unnecessary cost.

In the study setup, each model handled the full agent loop, including planning, background calls, and subagent work, without routing routine tasks to a faster model. Fivetran said this increased cost and end-to-end execution time when an agent took more steps before returning control to an engineer. The company did not measure potential savings from a mixed-model workflow.

Fivetran plans a follow-up blinded comparison of planners and orchestrators in a non-proprietary harness. It says the next study will hold execution models, tools, and budgets constant across predefined task categories while measuring task completion, reviewed code quality, human effort, latency, and cost. For users, the findings support choosing models according to the task rather than assuming the most expensive option will provide the best coding experience.