Replit's ViBench 2.0 tests AI models on building whole software products

What ViBench 2.0 measures

ViBench 2.0 joins the Specialized Intelligence Index (SII) from Fireworks, which Replit describes as its partner for open-weight model inference and training. The SII collects practitioner-built benchmarks to compare open, closed and specialized models on real work tasks across industries.

Replit contrasts the new benchmark with coding tests such as DeepSWE, which hand an agent a well-defined task inside an existing codebase. In ViBench 2.0 the agent begins with nothing. It builds a first version of a product from a short spec, then adds up to eighteen features on top of its own code. The products are modeled on services like Uber, GitHub and Google Docs.

The benchmark contains seventeen products, 241 feature requests and 544 test plans. Eight of the products are public on GitHub in the Harbor format, and the other nine are held out. Replit says all results can be rerun with one command from release v2.0.6 on its OpenHands-based harness.

The design builds on ViBench 1.0, which found that models score lower when they add a feature to their own code, a setting the paper called Vibe-on-Vibe. Replit says ViBench 2.0 extends this by emulating how a product evolves. Because the agent must choose its schema and routes before knowing what comes next, later features can be harder than earlier ones.

How the grading works

Feature requests describe what a user expects, not how to build it. Once the last feature is built, a grading agent runs the test plans in a real browser, clicking, typing and checking results. The tasks test things like clean deployment on an empty database, existing data, invariants, stale state and concurrency. One example: in the ride-hailing product, split-fare shares must add up exactly to the ride cost, to the cent.

Replit uses Opus 5.5 as the grader because it made the fewest mistakes among the models tested. To check it, the team planted 34 bugs one at a time in working builds, and according to Replit it never passed a plan while the bug was visible. When GPT-6.1 Sol re-graded 477 of Opus's passes, 99.2% were confirmed correct. Replit also says it saw no sign that Opus favors builds from Claude models.

One public test plan has two users, Alice and Bob, editing the same rectangle in a design tool, then checks that Alice's undo reverts only her own change. GPT-6.1 Sol, GPT-6 Astra and Opus 5.5 fail this plan in eight of their twelve builds.

Results and cost

The official results come from Fireworks' runs: four builds of each of the seventeen products per model, on the same harness and settings, at medium effort and temperature 1.0. Pass@1 is the share of builds in which every test plan passes.

• GPT-6.1 Sol leads with a pass@1 of 65%, followed by GPT-6 Astra at 60% and Opus 5.5 at 49%.
• All three pass more than 90% of their test plans, so most failed builds miss by only a few plans.
• Sol costs about $12 per build, and Astra about $90.
• DeepSeek V4.1 Flash passes 77% of its test plans for about $2.50 per build, though no open-weight model passes a full build.
• Opus 5.5 finishes a build in about 70 minutes on average, roughly 2.5 times faster than Sol. Replit attributes the gap mostly to API provider speed.

Replit also ran earlier GPT models itself on the eight public products. Pass@1 rises from 0% for GPT-5.4 to 31% for GPT-6 Sol and 53% for GPT-6.1 Sol, while plan pass rate climbs from 42% to 95%. GPT-6.1 Sol's 53% on the public set sits below its 65% overall. GPT-6.1 Terra, which Replit calls OpenAI's newest model at the time of writing, passes 47% of builds at about half the cost of GPT-6.1 Sol.

Where models fail and how builds look

The missed plans are often small details a real user would notice: a draft pull request that never shows its merge conflict, no way to comment on a word inside a table cell, or a server owner with no roles who cannot moderate anyone.

Replit identifies two failure modes. In about one in 40 builds from the top models, the product never starts on the empty database, because later schema code references a table before the statement that creates it. More often, builds break where features intersect. Nearly three-quarters of those models' failed builds fail only on such intersections.

ViBench does not grade design, but Replit shared screenshots anyway. It says models from the same family produce similar designs. GPT models tend to add illustrations and big headlines, while Claude models build plainer screens closer to the products they imitate. In Replit's view, the best-looking builds were no more likely to pass.

What it means

Replit concludes that today's best models build most of a product correctly, but that adding features one at a time exposes their limits, since new features must fit code the agent wrote earlier. The company expects scores like these to reach more cost-efficient models, including open-weight ones, and sees GPT-6.1 Sol at about $12 per build as an early sign. A detailed methodology write-up is coming, and current results are on the SII leaderboard and vibench.ai.