AI Taste Study: Spending 9.5 Times More Bought Only Four Points of Preference

How the study worked

The analysis builds on HumanSignal's AI taste study. Ten models from ten labs each built a site for the same ten briefs. People then chose between the results in blind pairwise comparison, with no model names on screen. According to HumanSignal, the study covered 450 matchups, 4,500 pairwise preference ranking annotations and 29 annotators.

Each run also carried an inference cost. The new post, written by data scientist Micaela Kaplan, asks whether higher spending produced work people liked more.

Nine times the cost, four points of preference

The headline comparison involves two models. muse-spark-1.3 built its sites for an average of $0.16 of inference and finished third overall with a 60.3% preference share. gpt-6-astra spent $1.53 per site, about 9.5 times as much, and finished first at 64.3%. In HumanSignal's words, that extra spend bought four points of preference and two places in the ranking.

Preference is the share of 900 blind pairwise votes each model won across the study. Other results from the table include:

• grok-4.6, at $0.345 per site, ranked second with 62.7%.
• gemini-3.8-flash, at $0.427 per site, ranked ninth with 32.8%, despite being the third most expensive model.
• seed-2.0-code, at $0.181 per site, ranked tenth with 19.1%.

Ranking the ten models by spend and by preference produced a correlation of +0.06. Within a single model, comparing its ten briefs, the correlation between spend and result averaged +0.03 across all ten models, and four of the ten were negative. Run time did not help either: runs took between 162 and 1,604 seconds, and the correlation between duration and performance was −0.17, so the slowest sites did slightly worse.

Cost buys a floor, not a ceiling

The data does show one pattern. No run in the study cost more than $0.90 of inference and scored below 35%. HumanSignal reads this as spending buying out the worst outcomes, since failures cluster among cheap and mid-priced runs. In the post's phrasing, "Cost buys a floor. It does not buy a ceiling."

Cheap is not a strategy either. seed-2.0-code lost roughly four of every five votes it appeared in. According to HumanSignal, this was largely because the model failed to generate a viable site for nearly every brief. The company says it cannot tell whether cost was the cause, calling that unlikely but unknowable. It also points out that muse-spark-1.3 and seed-2.0-code cost within two cents of each other per site yet finished third and tenth.

Why price does not predict taste

HumanSignal argues the result is expected. A price per token reflects parameter count, serving economics and what a lab decided to charge, none of which measures output quality. In much of engineering, more compute buys a better answer because correctness can be checked. Taste has no such criterion: "There is no compiler," the post says, so extra compute has nothing to optimize against.

What it means for choosing a model

For teams picking a model for generated design work, HumanSignal says neither a price list nor a capability benchmark will rank the options. All 100 sites in the study render, work and pass, so benchmarks of that kind do not separate them. The company's recommendation is to show the output to people and ask which they would choose, a slower and more expensive method that it calls the only one that answers the real question.

All 100 sites, their costs and the full preference data are available in HumanSignal's public dashboard.