A tournament for choosing among AI answers

Generating more than one answer gives a language model several chances to solve a problem. But the extra answers are useful only if a selection method can identify a good one. A confident-looking explanation can still contain a subtle mistake, and the most common answer is not necessarily the correct one.

In a preprint, Hannah Guan and Yilun Du study a different way to choose: put the candidates through a knockout tournament, comparing two at a time on several specific criteria. They call the method Aspect-wise Pairwise Tournaments, or APT. The method chooses from answers already generated; it does not create a new solution or guarantee that a correct one is present.

A panel of narrow comparisons

APT starts with a pool of candidate solutions. In each match, language-model judges see two candidates and compare them from the perspective of a particular aspect, such as logical soundness or domain knowledge. A verifier can express a preference or abstain. The candidate receiving more votes advances; a tie is resolved randomly. The process continues until one candidate remains.

The judges are prompted versions of existing models, not separately trained specialists. The researchers also present each pair in both orders to address the possibility that a judge favors whichever answer appears first. A single-elimination bracket needs seven matches for eight candidates, although each match involves multiple verifier queries.

The paper illustrates the idea with a counting problem. Two solutions agree on much of the reasoning, but one misses a factor needed to count distinct rotations. In the example, several aspect-specific judges favor the solution that includes the factor, while a units-checking judge has no preference. The point is not that every aspect will be useful for every question, but that different prompts can draw attention to different features of a comparison.

What the experiments found

The authors tested three generator models on five benchmarks: two competition-math sets, a text-only multiple-choice subset of Humanity's Last Exam, a chess move-validity task, and a chess move-quality task. Each method selected from eight generated candidates per problem. The researchers compared APT with choosing the most common answer, choosing by a reward model's score, and using the same aspect verifiers to approve or reject candidates individually.

APT had the highest reported score in 14 of the 15 generator-and-benchmark combinations. For example, on the HMMT mathematics set, the GPT-5.2 generator's selected answers were correct 39.78% of the time with APT, compared with 36.56% using individual aspect approvals and 26.88% using the most common answer. That benchmark contained 93 problems, so these percentages should be read in the context of a relatively small test set.

Chess move quality was scored using the Stockfish engine, in pawn units, rather than as a simple correct-or-incorrect answer. The paper reports especially large gains on this task. It also compares APT with a tournament using repeated generic pairwise judgments at a matched verification budget: the aspect-based version scored higher in the tested settings. These comparisons support the authors' argument for combining pairwise comparison with separate criteria, though they do not establish that the approach will win on every task or with every judge.

Preferences and trade-offs

A user can add a criterion and give its vote more weight. In separate experiments, the researchers added a preference for shorter mathematics solutions or for aggressive chess moves, reusing the original candidate pools. Increasing the added criterion's weight shifted selections toward shorter answers or more checks and captures. For the math tests, the paper reports that accuracy fell by at most three problems in any tested generator, dataset and weight setting. Steering changes which available answer is picked; it cannot provide an option absent from the pool.

There are practical limits. A single-elimination bracket can discard a strong candidate early, and model judges can share mistakes or be affected by presentation order. More comparisons also cost computation, and the study uses particular models, prompts and benchmarks rather than a universal test of answer quality. The extracted PDF text used for this report may omit figures and distort equations or tables. The work is a preprint, and its results are evidence about the tested setup, not a guarantee of reliable AI verification in general.