Better Questions, Better Legal Predictions? Inside LegalEvolver

The search question matters

Imagine someone borrows a phone and then runs away with it. A search for the elements of fraud alone may miss the question that distinguishes competing charges: what did the owner intend when handing over the phone? A more useful search would include the facts and ask how they bear on theft versus fraud.

That example captures the premise of a preprint by Xinyu Yang and colleagues at Renmin University of China. Their system, LegalEvolver, aims to improve legal judgment prediction: given a case description, an AI agent predicts applicable law articles, charges and a prison term. Rather than treating retrieval as a simple add-on, the researchers focus on the queries an agent writes while consulting legal sources.

The distinction is important. An agent may have access to statutes and precedents yet remain anchored to an early, mistaken hypothesis. Repeatedly searching for evidence of that hypothesis can overlook facts that would point toward a different charge or sentence.

Learning from contrasting attempts

LegalEvolver uses four retrieval tools covering law articles, precedents, judicial interpretations and sentencing guidelines. Its training begins with multiple attempts at the same cases. The researchers also rewrite selected queries to clarify the legal question, add relevant facts or redirect a search after a particular type of failure. Successful trajectories become examples for supervised fine-tuning, in which the agent learns from selected demonstrations.

The next step turns differences between attempts into rubrics, or explicit scoring criteria. The researchers compare trajectories for the same case that reached different outcomes: one might get both the law and charge right but miss the sentence, while another might fail on legal qualification. A separate language model proposes criteria based on differences in their queries; these are aggregated and tested against held-out trajectory pairs.

The framework organizes criteria by retrieval tool and reasoning focus, such as distinguishing charges or finding sentencing information. In reinforcement learning, the agent receives feedback not only on its final prediction and output format but also on the quality of individual searches. After a training round, new trajectories are used to revise and revalidate the rubrics. The revised criteria then guide the next round.

This is not independent legal discovery: the criteria are generated and scored by language models, and their validation checks whether they distinguish preferred from less successful observed attempts. A query associated with a correct outcome is not necessarily the sole cause of that outcome.

What the experiments show

The authors tested a Qwen3-8B-based agent on balanced sets of 1,000 single-defendant cases from CAIL2018 and 2,000 multi-defendant cases from CMDL. Together, the sets covered 201 distinct charge categories. Performance was measured using accuracy and macro-F1, a metric that gives categories equal weight rather than allowing frequent categories to dominate.

LegalEvolver achieved an overall score of 61.5, compared with 60.4 for SmartSearch, a general query-optimization baseline, and 59.9 for standard reinforcement learning. That overall figure averages accuracy and macro-F1 across the three prediction tasks and two datasets; it is not the percentage of cases in which every part of a judgment was correct.

Controlled comparisons support a role for the training design. With the same training data and optimization budget, keeping the initial rubrics fixed produced a lower overall score, as did removing rubric rewards or replacing specialized rubrics with a shared one. The paper's “w/o SFT” variant also scored lower, but it was the base model without any subsequent post-training. That comparison therefore does not isolate the effect of supervised initialization from the effect of reinforcement learning. The results support the combined approach in this setting, not the legal soundness of every learned criterion.

The gains should be read alongside the absolute results. Prison-term accuracy was 22.1% on CAIL2018 and 25.6% on CMDL, even for LegalEvolver. The model performed much better on article and charge prediction than on exact sentence prediction.

Evidence, not authority

Three language-model judges also rated anonymized query sets, placing LegalEvolver first on measures including specificity and awareness of competing interpretations. That is encouraging but not the same as an independent assessment by legal practitioners. The paper also used a language model to categorize query errors; the authors report reviewing AI-assisted analyses.

The benchmarks and retrieval materials concern Chinese criminal law, and the test sets were deliberately balanced across charges rather than designed to mirror their real-world frequency. Results therefore cannot simply be assumed to transfer to other jurisdictions, populations or deployment conditions. Predictions can also inherit errors and biases from case data, legal corpora and models.

The useful conclusion is narrower than a claim that AI can decide cases: better-targeted retrieval can improve a research system’s predictions. The preprint describes the system as research-only, not legal advice or a substitute for lawyers and judicial decision-makers. Any practical application would need jurisdiction-specific validation, privacy protections, bias assessment and meaningful human oversight.