ViBench 2.0 tests whether AI-built apps still work after the last feature

Testing the finished product

In the November 3 arXiv preprint ViBench 2.0: A Benchmark for Long-Horizon Software Building, Daniel Furman and colleagues at Replit describe an evaluation designed around successive feature requests rather than isolated coding tasks. The supplied source is version 1 of a preprint; its reported findings should not be treated as independently established facts or evidence of peer-reviewed validation.

The benchmark contains 17 full-stack web products, 241 feature requests and 544 test plans. Eight public products are modeled on services including Amazon, Discord, Figma and Uber. These are benchmark applications with selected requirements, not demonstrated replacements for those commercial services.

Each agent starts with an empty project and receives requirements one stage at a time in the same conversation. Later features must build on its earlier code. Only the final application is graded, so an early feature must survive subsequent changes.

How the evaluation works

Builders use a shared agent setup with terminal access, a file editor and a task list. Each stage allows up to 300 agent steps and two hours. Older conversation content is summarized when context exceeds 200,000 tokens. Builders cannot see the test plans and receive no grading feedback between requests.

After development, a separate agent prepares any necessary starting data. A browser agent then operates the application through Chromium, checking what a user can observe. Fresh containers and initially empty databases test whether the delivered code can install and start cleanly, rather than merely work in its development environment.

About 85% of test plans exercise features together. Examples include whether splitting a ride fare preserves driver earnings and whether competing edits behave correctly across users.

The paper reports two scores. For each build, plan pass rate is the fraction of scored test plans passed. The reported model score averages those rates over each product’s four builds, then equally across products. It is not the fraction passed after pooling every plan: products have different numbers of plans. Pass@1 assigns success only when every scored plan in a build passes, then uses the same averaging procedure. Success means meeting this benchmark’s requirements, not proving the software bug-free.

High scores, incomplete builds

An external Fireworks team ran nine models through the released evaluation pipeline, according to the authors. Each model built every product four times, producing 612 builds overall.

GPT-6.1 Sol recorded the highest all-plans success rate at 64.7%, alongside an average plan pass rate of 96.9%. GPT-6 Astra achieved a 98.0% average plan pass rate but 60.3% all-plans success. Opus 5.5 scored 93.7% and 48.5%, respectively.

Those results illustrate the benchmark’s central distinction: high average plan pass rates do not reliably translate into applications that pass every check. Among 84 failed builds from these three models that reached seeding, half failed exactly one plan and 62 failed no more than two.

The three tested open-weight models produced no all-plans successes across their combined 204 builds. That finding applies to these models under this setup, not to open-weight software agents generally. DeepSeek V4.1 Flash nevertheless achieved a 77.4% average plan pass rate, showing why the partial score remains informative.

The authors also report improvement across three Sol releases on the eight public products: all-plans success rose from 3.1% to 53.1%, while builder cost fell from $26.5 to $11.8 per build. This version comparison covers one provider and combines author-run experiments with the external team’s newest-model results.

The judge needs testing too

Both seeding and grading use Opus 5.5, which is also evaluated as a builder. The protocol accepts an initial passing grade but rechecks failed plans until three verdicts agree, using up to five valid grades.

The authors’ validation found four false passes among 477 audited passing plans, approximately 0.8%. In planted-bug experiments, none of 96 grades passed while the targeted bug was visible. These are bounded validation results, not guarantees: repeated grading addresses occasional false failures more directly than persistent false passes, and the studies used the authors’ reference builds.

What the results do not establish

Four builds per product leave broad uncertainty intervals: Sol and Astra’s all-plans intervals overlap, preventing a clear ranking between them. Nine products remain private to reduce training contamination, so outsiders can reproduce only the public subset.

The benchmark covers English-specified, PostgreSQL-backed web applications in one fixed feature order. It measures functional behavior, not comprehensive security, visual quality, or the finished application’s operating cost and speed. Different agent setups or user feedback could change results.

Evaluation is also expensive: the authors estimate roughly $85 per build for seeding and grading, separate from builder costs. The practical takeaway is therefore narrower than a universal model ranking: final-state testing can expose failures that high average test-plan scores conceal.