PersonaCodeBench tests whether coding agents meet users’ needs as well as pass tests

Beyond a correct patch

A coding agent can pass a software task’s tests while leaving its user without the explanation, permissions or delivery materials they requested. In the PersonaCodeBench preprint, Haixin Wang and colleagues propose evaluating those obligations alongside functional correctness.

The benchmark pairs coding tasks with simulated users whose recorded characteristics include technical background, interaction preferences and working constraints. Relevant attributes become concrete, task-specific requirements: protect pre-existing changes, wait for authorization, explain a repair using source evidence, or accurately report validation attempts. Only attributes converted into explicit obligations contribute to personalization acceptance; a descriptive personality trait does not automatically become a hidden grading condition.

The authors call this acceptance an operational measure of explicit requirement fulfillment, not a direct measurement of human satisfaction.

Requirements unfold, but the contract stays fixed

PersonaCodeBench contains 231 task–persona pairs drawn from Terminal-bench 3, DeepSWE, SWE-bench-Live, SWE-Together and SWE-bench Pro. Its public Lite release contains 100 pairs with 186 added requirement entries. An entry can combine several obligations.

The requirements, disclosure conditions and acceptance rules are fixed before evaluation. During a session, a controller manages which requirements have been disclosed, what actions are authorized and whether the interaction can advance. A language-model user simulator expresses permitted requests and feedback, but cannot independently invent grading conditions or grant unauthorized permissions.

An online evaluator assesses active obligations. Missing evidence can prompt further feedback, while satisfied progression conditions can permit the next request. Exact procedures differ across the benchmark’s execution profiles.

Final assessment separates two questions: did the delivered software satisfy the original task, and did the agent satisfy all mandatory personalized requirements? A “Joint pass” requires both, subject to budget and evaluation-validity conditions. Conversational approval or an intermediate positive judgment does not establish final success.

What the reported scores show

On Lite, the highest reported Joint pass rate is 48%, achieved by GPT-6 Astra at maximum reasoning effort. GPT-6.1 Sol at extra-high effort records 42%, and Claude Opus 5.5 at medium effort records 40%. These are results under the paper’s particular tasks, scaffolds, simulators and budget policies, rather than general ratings of coding ability.

The denominator matters. Every outcome other than a recorded pass contributes zero to the headline rate, including evaluation errors and unexecuted tasks. Consequently, a low score can combine unsuccessful work with incomplete evaluation coverage; it is not necessarily a failure rate among fully graded sessions.

Results also vary substantially by task source. The best reported Terminal-bench 3 result is four successes out of 23 tasks, or 17.4%. Different configurations lead other source subsets. SWE-Together contributes only four tasks, making its percentages particularly sensitive to individual outcomes.

Added obligations versus delayed disclosure

An ablation using GPT-6 Luna compares the default workflow with two alternatives. Giving all added requirements in the opening changes Joint pass from 19% to 20%. That single-run comparison does not establish statistical equivalence, but offers little aggregate evidence that earlier disclosure improves performance in this setting.

Removing the persona profile, added obligations and their associated interaction machinery raises native-task success to 45%, compared with 24% completion and 19% Joint pass in the default setting. Because several components are removed together, this comparison cannot isolate the contribution of any one requirement or control mechanism.

Selected trajectories illustrate the practical distinctions. One agent passes native tests but fails personalized acceptance because its contribution record mishandles existing workspace content and omits earlier failed validation attempts. Another workflow verifies rollback but leaves no remaining task changes for the objective checker. These cases concern evidence and final workspace state, not merely conversational tone.

Limits of the measurement

The collection was deliberately screened for tasks a reference agent could solve natively but struggled with after persona adaptation. It therefore should not be treated as a representative sample of everyday coding requests.

Automated assessment is also imperfect. The paper reports a blinded review of 148 requirement-level judgments by two software-engineering experts, followed by adjudication. After adjustment for stratified sampling, three-way agreement with the expert reference is 71.0%; agreement on an accept/reject decision is 77.2%.

Other audits cover narrower properties, such as format-checker behavior and preservation of disclosed text. They do not establish universal semantic accuracy. PersonaCodeBench’s contribution is an inspectable way to distinguish correct software from explicitly compliant delivery—not proof that simulated acceptance captures what real users would value.