AI design benchmark separates finished runs from verified delivery

Following requirements through revisions

DRAFT-Bench evaluates AI agents across requirement discovery, creative exploration, editing, revision and production handoff. Introduced in a preprint by Rong Pan and Jianqiang Wang, it distinguishes completed execution from compliant delivery and professional acceptance. This report draws on the supplied full paper text, not only its abstract.

The primary comparison tests Codex/Astra and Claude Code on 35 design briefs under 43 execution conditions. Assignments include packaging, brand identity, e-commerce and production handoff. Some briefs appear in multiple contexts, so these conditions are not independent client projects. The assignments are synthetic or curated, rather than observations of customer deployments.

A separate track contains 40 simulated request-for-quotation, or RFQ, scenarios. These assess requirement gathering but are not connected to the design assignments. The study therefore does not test a complete journey from client questioning to accepted delivery.

Connecting instructions to files

The authors’ Requirement-to-Artifact framework asks which requirement is active, which output version should satisfy it, what must change and what must remain protected. It separates observable file properties from an agent’s assertions, automated judgments and professional acceptance.

That distinction matters when feedback changes the scope of work. Selecting a packaging direction might activate an earlier obligation to produce a complete product family. Conversely, a later instruction might assume that a previous version exists when the agent never produced it.

The researchers audited archived semantic checks for every scored core condition and directly inspected eight selected pairs of outputs. Those inspections illustrate possible mechanisms and ambiguities; they cannot establish their frequency. Gemini 2.5 Pro supplied the archived design judgments. Independent expert acceptance was not measured.

Completion is not compliance

Astra completed all 43 design conditions. Claude completed 35, with eight first-turn timeouts under a 60-minute active-call limit per turn. This was not a total-project time budget.

Every scored core run received full structural credit. Nevertheless, among the 28 core conditions scored for both systems, automated semantic checks flagged at least one violation in 11 Astra conditions and five Claude conditions. Structural checks address properties such as file presence, readability, dimensions and format; semantic checks address task-specific obligations.

These are model-judged flags, not expert-confirmed failure rates. An unflagged output is not certified acceptable. The shared comparison also excludes four Claude core timeouts, so examining only scored outputs leaves out part of the execution record.

Claude had a higher observed weighted quality contribution on that shared subset. But missing rubric measurements produced overlapping full-score bounds: 0.711–0.968 for Claude and 0.655–0.912 for Astra. The missing evidence permits either overall ordering. These ranges are not statistical confidence intervals.

Where delivery becomes ambiguous

In one packaging assignment, feedback selected a strategy and requested a hue adjustment. Astra revised a two-product sample; Claude exported five individual product images and a family lineup. The original brief requested five products after selection, but a runner instruction emphasized applying only the requested change. The authors leave that scope conflict for expert adjudication rather than treating file counts as a complete quality verdict.

In a production-handoff case, Astra supplied a reference proof and a PDF marked not for printing, then requested an approved source. Claude exported a revised PDF containing a White ink separation resource. The authors’ inspection established that resource’s presence—not correct plate placement, print suitability or factory acceptance.

Other inspected cases contained missing revisions that both systems explicitly disclosed while requesting clarification. These remained unresolved deliveries, but did not by themselves demonstrate false completion claims or establish whether stopping was justified.

The evaluator could also interpret obligations differently. In one e-commerce case, both systems received an explicit direction selection, yet their archived complete-set checks used different applicability decisions. The discrepancy remains unresolved.

Strong intake, separate evidence

RFQ simulations used fixed customer replies, seven requirement fields, four critical fields and at most eight replies. Success required eliciting at least 80% of initially unknown fields and all initially unknown critical fields.

Across 39 jointly scored scenarios, average elicited coverage reached 98.9% for Astra and 98.2% for Claude. Success counts were 36 and 34, respectively, but the ordering reversed between the two cohorts. Auxiliary models and attempt histories differed, and the original cohort included a disclosed post-hoc reattempt policy. These results do not establish general superiority in questioning or downstream design success.

Limits on the comparison

The design systems had different tools: Astra had built-in image generation, while Claude used code-based visual production without an image-generation model. Runtime and environment differences further prevent attributing results solely to model capabilities. Each design condition had one included execution, leaving repeat-run variability unknown.

Public materials include briefs and numerical records, but omit complete candidate outputs, judge payloads and other evidence needed to recreate the design judgments. The supported conclusion is methodological: execution, requirement compliance and professional acceptance require separate evidence.