AI debate can become training data—but agreement is not proof

Training a language model to reason usually requires some way to identify good answers: worked examples, answer keys, executable checks or another model acting as evaluator. In the preprint Alignment by Confident Debate: Preference Learning from Multi-Agent Debate, Nikhil Verma and colleagues at Upwork investigate an alternative: constructing training preferences from models debating one another.

Their method, AbCD, separates two questions that voting tends to combine: which responses give equivalent answers, and which answer deserves to win? The researchers report useful improvements on mathematical and quantitative reasoning benchmarks. Extensions to programming and medical questions use additional evaluation tools, however, so the whole study should not be read as demonstrating learning without external supervision in every domain.

How disagreement becomes a lesson

The default committee contains five copies of the same instruction-tuned model. Different sampling temperatures make their outputs more or less variable. Each copy first solves a problem independently, then receives the others’ solutions and produces a revised response. This is one initial generation followed by one exchange round—not a prolonged argument.

For mathematical questions, the system extracts each final number or answer-choice letter and groups equivalent answers together. Different explanations ending in the same numerical value belong to the same group.

It then scores each response using the probabilities the model assigned to the tokens in its final answer. Specifically, it calculates their geometric mean, a length-normalized measure of how likely those answer tokens were during generation. This is not a verified probability that the answer is correct.

The scores are added within each answer group. The group with the greatest combined confidence supplies the preferred responses; competing responses become dispreferred examples. Thus, the method resembles a confidence-weighted vote, rather than choosing either the largest group or the single most confident response automatically.

Standard training algorithms then use these labels. Direct Preference Optimization, or DPO, contrasts preferred and rejected responses to the same question. Supervised fine-tuning, the comparison method, imitates only the preferred responses. Neither requires the mathematical answer key to construct these training labels, although the researchers use answer keys to measure performance.

What improved—and what the numbers mean

The main mathematical and quantitative reasoning comparison covers four models across GSM8K, SVAMP and MathQA. In all 12 model–dataset combinations, the best preference-training method improves consensus accuracy over inference-only debate. DPO produces the best post-training result in nine combinations.

The largest improvement in that main comparison is for Llama-3.2-1B on SVAMP: consensus accuracy rises from 19.0% after debate to 40.5% after DPO, an increase of 21.5 percentage points. Other gains are much smaller. For Phi-4-mini on SVAMP, the best result moves from 89.7% to 90.0%.

These are committee-level results: evaluation still runs multiple agents and selects an answer after debate. They do not mean a single model response achieves those same accuracies. Individual agents also improve in many settings, but their average accuracy is a separate measurement.

Confidence weighting alone offers a modest overall advantage over ordinary voting on some tasks. Across four models on GSM8K, average answer-selection accuracy is 57.5% for confidence weighting and 57.2% for majority voting. Its advantage is more pronounced on tied or disputed cases, and it does not win in every model–dataset setting.

The extensions use different signals

The programming experiments require an important qualification. Candidate programs are grouped by their outputs on public test inputs, and groups are ranked using public tests passed—not answer-token probabilities. Additional challenge assertions, where available, are withheld from selection and used in evaluation. This extension therefore uses an executable checking signal.

An exploratory medical-question experiment adds another dependency: GPT-4o judges whether responses satisfy supplied checklist criteria. Responses are grouped by their checklist patterns and ranked by checklist coverage. Reported improvements are modest, and complete checklist success remains rare. This is evidence about judge-scored responses, not proof of clinical reliability or a judge-free solution to open-ended reasoning.

Why confidence still needs caution

Longer debate often increases agreement without improving correctness. The paper includes examples where initially correct agents adopt a shared wrong answer. Giving greater training weight to strongly supported winners does not improve the tested DPO configuration.

Other limits concern experimental coverage. Most main results use two training runs, and many evaluations use dataset subsets. Larger-split experiments support the direction of improvement for one model, but repeated self-training does not consistently add gains. Increasing the generation limit also changes baseline accuracy and reduces some apparent advantages, showing that answer truncation matters.

Finally, confidence scores from different model architectures are not necessarily comparable. A calibration experiment uses gold correctness labels to rescale them, adding supervision back into that variant. AbCD’s central finding is consequently narrower than automatic self-correction: under suitable parsing and scoring conditions, competing model responses can supply useful training preferences—even though confident consensus can still be wrong.