When Can Fast Decision Models Replace LLM Judges?

AI systems increasingly judge other AI systems, selecting answers, filtering training data and scoring responses. In a preprint, Scale AI researcher Shreyas Subramanian investigates when models that return decisions without generating text can take over this work—and when a reasoning model is still needed.

Decisions without generated explanations

An AI judge does not always need to write an explanation. Sometimes its job is simply to choose the better answer or assign a score. The preprint examines System-1 decision models, including TypeSafe’s Jev, which answer structured questions with probabilities in a single forward pass rather than generating text.

Subramanian evaluates 14 such models across eleven established benchmarks, comparing them with Claude Haiku 5.5, GPT-5.6 with and without reasoning, and Claude Opus 5.5 with thinking. The tasks cover correctness, instruction following, human preferences, answer selection, long inputs and web-agent trajectories.

Models were selected using a stated protocol combining leaderboard-based choices with an API catalog screening process. Pairwise comparisons were evaluated in both presentation orders, with probabilities averaged to reduce order effects. Statistical comparisons used paired tests and corrections for multiple comparisons.

Model choice matters

The results do not support treating decision models as interchangeable. On JudgeBench, pairwise accuracy ranged from 66.6% for Kev-4B to 84.6% for d1-ultra. On RewardBench 2’s best-of-four task, scores ranged from 57.3% to 87.0%.

Five models—DReX-Max, d1-ultra, Mercury-Decide, Jev 1.14 and pplx-decider 1.2—showed no statistically significant accuracy difference from GPT-5.6 without reasoning across all five headline pairwise and best-of-four benchmarks. This means the tests did not establish a difference; it does not prove the models are equivalent.

DReX-Max and d1-ultra significantly outperformed Haiku on JudgeBench, LLMBar and RewardBench 2, by 10.5–14.4 percentage points. Checklists, which assessed responses through multiple yes/no criteria, mainly helped weaker models rather than consistently improving the strongest ones.

A shared reference changes the task

The largest improvements came from generating a reference answer once per prompt using GPT-5.6 with reasoning. Decision models then checked whether each candidate’s final answer agreed with that reference, rather than independently solving the original problem.

On JudgeBench, eleven models reached 90.3–93.4% accuracy with this approach. Their differences from GPT-5.6 with reasoning and Opus with thinking were not statistically significant under the study’s corrected tests.

On 180 single-answer prompts from PPE, each with sixteen candidate responses, reference-guided models selected a correct answer at rates of 87.2–92.8%, compared with 91.2% for GPT-5.6 with reasoning. Sharing the reference spread its generation cost across all sixteen candidates. The reported Jev pipeline cost $0.0051 per prompt, versus $0.0827 for direct scoring by GPT-5.6 with reasoning—approximately one-sixteenth as much.

This is a hybrid workflow, not reasoning-free verification. It also depends on task structure: on RewardBench 2, reference-guided grading was less accurate than checklists for eleven models because many items lacked a single checkable answer.

Confidence is useful, but not protection

Calibration measures how closely stated probabilities match observed outcomes. Jev had the lowest reported expected calibration error, 0.019, compared with 0.038–0.087 for the generative judges. Three other decision models also scored below every generative comparator.

Useful confidence estimates supported selective escalation. On the LLMBar test split, pplx-decider followed by Opus for uncertain cases achieved the same 91.7% accuracy as Opus alone while escalating 17% of items. Routing was less successful on some other tasks, where thresholds chosen on development data did not transfer well.

Good calibration did not guarantee resistance to manipulation. An appended note falsely claiming that an incorrect answer had been verified reversed 0–4.7% of initially correct verdicts for five decision models, but 16.7–49.0% for the other nine. Jev’s rate was 23.3%. The author therefore argues that injection resistance must be measured for each model and judging formulation.

Where the evidence stops

The study reports single-run, mostly pilot-scale evaluations. Some higher-ranked decision models were excluded because of serving, hardware, cost or request-limit constraints. Injection tests used the researcher’s own perturbations, not an established injection benchmark.

The strongest long-context decision-model results still trailed GPT-5.6 by roughly 10–12 percentage points. Cost estimates used list prices and an assumed GPT-5.6 pricing tier; latency measurements were not controlled comparisons.

Same-size experiments also found that enabling reasoning substantially improved a four-billion-parameter generative judge. Subramanian interprets this as evidence that the small decision model’s correctness deficit primarily reflected missing reasoning rather than size alone. Overall, the preprint supports task-specific replacement and hybrid judging—not a universal substitute for reasoning judges.