Can AI Improve Its Writing by Changing How It Judges Itself?

A language model can check a calculation against a known answer, but what counts as a better story or a more helpful explanation? Without an automatic correctness test, reinforcement learning risks teaching models to please an evaluator rather than improve their answers.

Sumeet Ramesh Motwani, Tejas Krishnan and colleagues investigate this problem in Rubric Curriculum RL: Exploiting the Generation-Verification Gap in Non-Verifiable Domains. The arXiv manuscript was submitted on November 4, 2026; its landing page reports acceptance at ICML 2026. The findings discussed here are the authors’ reported experiments, not independently replicated results.

Turning recognition into training

The starting idea is the generation-verification gap: a model may recognize a stronger answer among several candidates even when its typical answer is weaker. In creative-writing experiments, the authors generated multiple stories and asked the same model to select among them. An external GPT-5.2 judge generally preferred these selections over single-sample outputs, with larger benefits as the candidate pool grew.

That establishes a useful selection ability under this evaluation setup—not an objective measure of literary merit. The harder question is whether the selection signal can improve the model itself.

A fixed scoring checklist can become a target for exploitation. The paper’s static-rubric baseline earned increasingly high training rewards without consistent improvement in separately evaluated story quality. The authors interpret that divergence as reward hacking: learning shortcuts that satisfy the scoring system rather than its intended purpose.

A checklist that grows

Their method, Rubric Curriculum RL, or RcRL, changes both the feedback and its timing.

First, a frozen copy of the initial model compares pairs of newly generated answers, judging one rubric criterion at a time. Each answer receives a reward based on how often it beats the other answers. Unlike an absolute score, this asks which response is better without requiring a calibrated numerical quality scale.

Second, the training checklist expands gradually. Creative-writing training begins with prompt alignment, then adds criteria such as plot coherence, character motivation, pacing and originality. A new criterion arrives every 100 training steps. Once multiple criteria are active, the newest receives half the reward weight; earlier criteria share the remaining half.

The judge itself does not improve during training. What changes is the generated material and the emphasis of the reward. The intended benefit is to redirect learning before prolonged optimization of one target starts damaging overall quality.

This is not learning from nothing. The writing setup uses 200 synthetic prompts generated by having the base model analyze existing short stories, plus a hand-specified rubric. The narrower claim is that training does not require newly collected human preference labels or a stronger teacher model. Stronger models and human annotators are used for evaluation.

What the experiments found

The main experiments train the instruction-tuned Qwen3-4B model. On FictionBench, RcRL achieved a reported 73.9% win rate against the starting model, compared with 57.5% for the static absolute-scoring baseline.

Human evaluation provides another perspective. Three professional annotators each assessed 500 paired comparisons, with model identities hidden and presentation order randomized. Their overall preference favored RcRL 70.5% of the time. Agreement was moderate, with Fleiss’ kappa of 0.548, underscoring that these judgments are not interchangeable or definitive.

On Creative Writing Bench v3, pairwise win rates favored RcRL under all three external judges tested. But not every metric improved: Claude 4.5 Opus assigned RcRL a slightly lower absolute rubric score than the base model, despite preferring it in pairwise comparisons. The distinction matters because relative preference and checklist scoring measure different things.

The authors also report continued improvement beyond 1,000 training steps and better preservation of policy entropy, a measure of uncertainty in token choices. That supports their account of less restrictive generation, although entropy alone does not establish meaningful creative diversity.

For health conversations, RcRL used 13 general criteria introduced every 50 steps. The paper reports HealthBench score improvements over the base model of 14.60% on the full benchmark and 25.38% on its consensus subset. These are reported benchmark-score improvements, not clinical success rates. A curriculum using absolute scores captured most of those gains, suggesting that pairwise feedback contributes differently across domains.

What remains unresolved

The method still depends on imperfect judges and manually chosen criteria. Comparing every pair becomes expensive as the number of generated answers grows. Its theoretical analysis establishes local improvement under idealized assumptions, not guaranteed progress throughout practical training.

The evidence supports a bounded conclusion: changing how a model compares its answers, and gradually changing what it rewards, improved performance in these writing and health-conversation experiments. Whether the same approach reliably improves long research or planning tasks remains future work.