AutoRefine adjusts AI code reviewers using software workflow outcomes

AI coding tools can produce changes whose quality is difficult to establish through automated tests alone. One response is to use another large language model as a judge, screening changes before they reach human reviewers. But that judge’s instructions need checking, and instructions fitted to yesterday’s workflow may not suit today’s.

Researchers at Meta Platforms propose AutoRefine, a system that adjusts a code judge’s prompt using outcomes already recorded during development. Their paper, submitted to arXiv on November 6, 2026, is a preprint prepared for submission to the CAIN 2027 industrial track; the supplied source does not establish peer-reviewed acceptance.

Learning from outcomes without treating them as truth

AutoRefine uses labels such as whether tests passed or a proposed change was merged or abandoned. These outcomes are convenient, but they are not interchangeable with code quality. A change might be abandoned because it duplicates other work, while a failure might depend on a hidden test.

An “informed teacher” therefore examines training examples before calibration. It sees the same material available to the deployed judge, plus the recorded outcome. It must justify each label it keeps using that visible material. When it cannot explain the outcome, it abstains and removes the example from training. It can also correct clearly contradicted labels, although corrections were rare in the reported experiments.

The calibration loop then runs the judge, examines disagreements with retained labels and rewrites its prompt. New reasons to reject changes must be balanced with reasons to accept them, discouraging indiscriminate rejection. A separate validation set, which the teacher does not filter, selects the final prompt. The experiments allowed up to five iterations and replayed recorded code changes rather than rerunning the coding agent.

Better agreement when reliable evidence is available

The first experiment used an industrial codebase with execution-based outcomes. The judges compared an agent’s changes with production reference patches, with tests removed from both. Because the labels were reliable and reference patches supplied additional evidence, this experiment did not use teacher filtering.

On an 85-task test set containing 13 failures, calibration improved a rubric-based judge’s Scott’s π from 0.36 to 0.60. This statistic measures agreement after accounting for agreement expected by chance; it is not a percentage accuracy score. Adjusting the original judge’s score threshold alone did not match the improvement.

The calibrated rubric judge identified 64% of failures, and 68% of the changes it flagged as failures actually failed. It correctly accepted 95% of passing changes. Results were averaged over three replays, excluding roughly 5% of tasks per replay because the judge timed out or errored.

A separate pass-or-fail judge caught more failures after calibration but also made more false alarms. Its calibrated result came from only one replay, making that comparison less extensively repeated.

Missing information cannot be repaired by wording

Removing the reference patch exposed a different problem. In a retrospective check of the validation set, the teacher abstained on 12 of 13 failures and none of 72 passes: the patches looked correct without the hidden tests.

Two engineering teams also struggled when reviewing 56 selected tasks without outcomes or teacher judgments. They missed nearly all failures. This supports the interpretation that some examples lacked sufficient visible evidence, rather than simply needing a better judge prompt. It does not establish that abstention always identifies such cases correctly.

In a controlled noise experiment on a performance-optimization agent’s calibration data, the researchers randomly flipped about one-quarter of outcome labels. Teacher filtering reduced the mislabeled share from 24.9% to 20.4%. However, the filtering was asymmetric: it retained most genuinely failing changes that had been mislabeled as passes. Cleaner-looking failures can therefore survive the filter.

Improvement depended on the workflow

Recalibrating the performance agent’s judge improved agreement on 17 independently held-out changes from its current workflow: Scott’s π rose from 0.16 to 0.51. The judge accepted all eight changes that ultimately landed, compared with six before recalibration. It identified five of nine abandoned changes, versus four previously.

These are small counts. The reported 95% confidence interval for the agreement gain extended from 0.00 to 0.74. A larger, 43-change analysis also showed improvement, but included examples used to select the prompt and was therefore not a wholly independent test.

On 16 older changes, the previous judge performed better: agreement fell from 0.50 to −0.02 after recalibration. All studies came from one organizational source, the workflow study involved one agent, and teacher and judge shared a model family. Each calibration loop ran once per setting. The findings support workflow-specific recalibration, not a universal code-quality guarantee.