Veritas gives AI-written task checkers a clearer view of the evidence

An automated improvement loop needs a way to distinguish progress from failure. If its checker rewards incomplete work or rejects valid solutions, the loop can select the wrong changes. Writing those checkers by hand takes effort, especially when tasks involve software, customer-service conversations or scientific workflows rather than a single numerical answer.

In a preprint, Daoan Zhang and colleagues introduce Veritas, a framework for generating executable task checkers, or verifiers. Their central finding is that the instructions and tools surrounding a language model can substantially affect the reliability of the checks it writes. The study measures agreement with benchmark judgments; it does not demonstrate an agent becoming more capable through repeated self-improvement.

Veritas builds on the OpenCode coding-agent framework and adds three components. First, an evidence header states exactly what the eventual checker will receive. A checker examining a finished filesystem cannot assume it will also see the agent’s conversation. Conversely, one receiving database snapshots should use those records to establish whether a requested change actually happened, rather than accepting a claim in the dialogue.

Second, a searchable library supplies domain rules that a task description may leave unstated. Its 19,603 entries cover financial, legal, airline and banking contexts. This component is activated when the checker receives a record of the agent’s actions or conversation. The agent is instructed to select relevant rules rather than apply the whole library indiscriminately.

Third, nine authoring principles tell the model to check observable outcomes, tolerate equivalent correct solutions, reject empty or superficial work, and deliver a usable checker. The aim is to avoid both excessive leniency and excessive strictness. An exact wording requirement, for example, can reject a correct response when the task never prescribed particular words.

The researchers compared Veritas with seven general-purpose agent frameworks while keeping the underlying model, GLM-5.2, fixed. For four execution-scored benchmarks, checker generation took place without access to candidate solutions, official answers or official tests. The generated checkers were subsequently evaluated on candidates including reference solutions, empty attempts and deliberately altered outputs.

Because official benchmark checkers can also be wrong, the team screened tasks and candidates before using their verdicts as labels. The screening included checking reference solutions and investigating suspicious judgments, although not every benchmark received every screening step. This makes the comparison more informative, but does not turn the retained labels into an exhaustive guarantee of correctness.

Veritas led the seven plain frameworks on all four execution-scored benchmarks. Its headline agreement scores were 0.787 on the conversational τ²-bench, 0.720 on SWE-bench Verified, 0.465 on Terminal-Bench and 0.548 on TB-Science. The strongest plain framework scored 0.726, 0.693, 0.387 and 0.474 respectively. These are chance-adjusted agreement scores, not percentages of tasks solved. The metric also counts a missing verdict as an error, so successfully delivering a checker contributes to the result.

That distinction matters. On Terminal-Bench, some frameworks produced reasonably accurate judgments when their checkers worked, but failed to provide verdicts for many candidates. Veritas left seven of 189 candidates without a verdict. Nevertheless, its reported reference-solution acceptance rate was only 45%, with missing verdicts counted as errors. Better performance than competitors therefore still left substantial false rejection.

The benefits were not universal. Adding Veritas’s components to two other frameworks improved five of eight execution-scored comparisons. With other underlying models, Veritas beat the corresponding vendor framework across all four execution benchmarks for Qwen3.7-Plus and Gemini 3.5 Flash. With GPT-5.6-sol, however, Codex CLI performed better on the three reported execution benchmarks. On two response-comparison benchmarks, Veritas was at or near the best, but the checker-writing agent could see both responses, making these a different evaluation setting.

There are important limits to the evidence. The main conversational and software evaluations used tasks involved in developing the framework; the two terminal benchmarks were introduced after its mechanisms were frozen. Most candidates were constructed rather than drawn from diverse real agent failures. Configurations generally ran once, and reported repeats showed variation large enough to warrant caution about small margins.

The practical result is narrower than a solution to automated self-improvement: explicit evidence boundaries, relevant rules and careful comparison criteria can improve AI-written checkers. Whether those improvements reliably select better changes inside a working improvement loop remains untested.