AI literature reviews: better access helps, but expert judgment still matters

A review with a known reference point

Abdullah Al Kalbani and colleagues evaluated six tools using a published review of the Spanish plume, a weather pattern associated with severe thunderstorms in western Europe. Its underlying collection contained 102 peer-reviewed articles published between 1968 and 2023.

The benchmark distinguished 18 articles that discussed the topic substantially from 84 that mentioned it only in passing. It also documented ambiguity: researchers have used “Spanish plume” to mean an elevated airstream, a broader weather pattern, or both. That made the collection a test of interpretation as well as retrieval.

The researchers assessed article discovery, interpretation, synthesis and proposed research directions. Not every tool performed every task. The benchmark review and its classifications were withheld from the tools, although the authors could not rule out exposure during model training.

Access changed what tools found

In the initial scored searches, five tools collectively named 57 benchmark articles. Codex found the most individually, with 42, followed by ChatGPT Deep Research with 40. Gemini Deep Research named 16, Claude Code 15 and Claude’s web interface seven.

These are not clean model rankings. Login and browser conditions differed across the initial runs. Gemini also reported 19 additional articles without naming them, preventing verification against the benchmark.

A paired experiment showed how strongly the search environment mattered. After university library login, Codex’s discovery increased from 42 to 82 articles and Claude Code’s from 15 to 79. Authentication provided subscription access and helped overcome some website blocks; the experiment could not separate those effects.

Other obstacles included automated-access restrictions, databases that searched only titles and abstracts rather than article bodies, and older material available as scanned images. The results therefore reflect publishing infrastructure and search tools, not just language-model capability.

Recognizing a mention was easier than judging importance

For the remaining tasks, Codex, Claude Code and NotebookLM received all 102 PDFs, removing discovery gaps from the comparison.

All three correctly identified the 18 substantial articles. But they also promoted many passing mentions into that category: Codex misclassified 66 of 84, NotebookLM 65 and Claude Code 34. In this experiment, finding every substantial article came with substantial overinclusion.

Quotation checking revealed a different pattern. Every quotation supplied by Codex and Claude Code in their structured databases matched wording in the cited article, although Claude Code gave three incorrect page numbers. NotebookLM altered wording in seven of its 101 quotations and frequently supplied incorrect page locations. Accurate copying and sound interpretation were separate abilities.

Mostly accurate did not mean insightful

The three tools then wrote syntheses organized around ten prescribed themes. Two domain experts evaluated anonymized versions without knowing which tool produced them. One expert had led the benchmark review; the other had not participated in it.

The authors report broadly accurate summaries but limited insight, alongside unsupported claims and exaggerated statements of agreement. One synthesis repeated a description found in several articles that conflicted with the original Spanish-plume definition. This illustrates why a real citation can still support a misleading account of a field.

The reviewers differed substantially in their scores and tool rankings. Nor did broader citation coverage guarantee better evaluations: Claude Code cited 79 articles, NotebookLM 41 and Codex 27.

For future research, Codex and Claude Code proposed questions resembling recommendations in the benchmark review. All three, however, also suggested topics broader than the Spanish plume itself.

A bounded result, not a universal verdict

The study tested one subject, one run per tool and task, and versions available in July and August 2026. It did not evaluate specialized literature-review systems. Interpretation scoring covered only four of 17 database fields, treating the human benchmark as correct.

The synthesis prompt also supplied the themes and emphasized avoiding errors. The authors acknowledge that this design may have discouraged interpretation and contributed to limited insight. Alternative prompts were not tested.

Their practical recommendation is to use AI assistance with an audit trail: check access barriers, search separately for missing publishers and older work, judge article importance independently, and verify claims and quotations against original sources. These experiments support that cautious workflow—not a claim that every AI tool will fail on every literature review.