Can sample answers teach AI what a good answer looks like?

The challenge: rewarding answers without a checklist

Reinforcement learning with verifiable rewards works when a task has a reliable checker—for example, one that determines whether a calculation is correct. But many tasks, including scientific writing, have no equally clear test. A useful abstract may need to be coherent, relevant and appropriately specific, qualities that can be hard to capture in a hand-written scoring rule.

The paper asks whether reference answers alone can provide a training signal. Its method, Reinforced Adversarial Learning (RAL), trains two language models together. One generates answers; the other learns to classify whether an answer came from a reference collection or the generator. The judge’s estimate that an answer is from the references becomes the generator’s reward. The judge sees the prompt and final answer, not the generator’s reasoning trace. The generator is trained with reinforcement learning, without example reasoning steps to copy.

How the researchers tested it

The researchers first tested RAL on integer multiplication, arithmetic-expression generation written as code, and converting chemical SMILES strings into molecular formulas. These tasks have objective correctness checks, but those checks were withheld during RAL training and used only for evaluation. This let the team test whether a judge trained on examples could learn a reward that tracked task performance.

They then studied two kinds of scientific writing. In one, the generator received a generic request to write a credible research abstract, while its reference examples came from medical and biomedical papers. In the other, it wrote U.S. National Institutes of Health (NIH) grant abstracts based on proposal titles. The researchers compared generated abstracts with real ones using language-model judges and, for the NIH task, human scientists. The trained generators and co-trained judges were Qwen3 models. In the NIH experiment, the generator had 32 billion parameters and the judge was a much larger model. Other language models, including Claude Haiku and frontier-model evaluators, were used for comparison or assessment. The authors say they release code and training configurations.

What the authors report

On the three checkable tasks, RAL with sufficiently capable judges and enough training reached results comparable to direct training with verifiable rewards. Results varied by task and judge size: larger judges generally helped, and the largest judge tested on the code-based arithmetic task exceeded the verifiable-reward baseline in the reported evaluation. On the chemistry task, supervised fine-tuning on reference examples also performed strongly, suggesting the advantages of RAL can depend on the task.

For biomedical abstract generation, RAL shifted outputs toward the reference domain while preserving semantic diversity, according to the authors’ measures. A comparison method that optimized against a fixed judge increased its reward but produced a narrow, off-target theme. In the NIH experiment, RAL-generated abstracts were more often mistaken for real abstracts than outputs from the tested frontier-model baselines. The reported fooling rates were 35% for an ensemble of language-model judges and 55% for human evaluators. These figures measure resemblance in the study’s comparison setup—not whether generated proposals were factually correct, scientifically sound or suitable for funding.

What remains uncertain

RAL’s central dependency is also a practical limitation: the judge must infer useful qualities from examples. The authors found that judge capacity mattered, and some settings required a judge much larger than the generator. That adds training cost, even if the judge can be discarded afterward. NIH results also fluctuated during training, which the authors identify as a stability challenge.

More broadly, matching a reference distribution is not the same as verifying truth. A judge may reward features that make text resemble its examples without ensuring that its claims are accurate. The paper’s frozen-judge comparison illustrates how a model can increase its score while missing the intended target. The study covers particular models, datasets and training choices; it does not show that RAL will work reliably across scientific fields or other high-stakes uses. Some writing evaluations rely on language-model judgments, while human comparisons measure whether text is distinguishable from real abstracts—not whether its claims are valid.

The findings are evidence that a co-trained judge provided a useful training signal in these experiments. They do not show that sample-based judging can replace expert review or factual verification. The work is available as an arXiv preprint; the supplied material does not establish its peer-review status.