An AI report can copy the structure and still get the facts wrong

A recurring report poses a deceptively difficult writing task: keep the familiar organization, but replace its contents with new evidence. A quarterly financial review should still look recognizable next quarter, without retaining outdated figures or dates. A literature survey faces a similar challenge when new papers arrive.

In the FRAME preprint, submitted to arXiv on November 6, 2026, Nan Jiang and colleagues investigate this task through 120 benchmark instances. Their central finding is that reproducing a report’s structure and producing a complete, evidence-supported update are different abilities. A document can perform similarly to another on organization while doing much worse on substance.

Building an update task

The benchmark contains four equally sized categories: quarterly financial reports, annual financial reports, and academic literature syntheses using smaller or larger paper collections. Sources come from the SEC’s EDGAR filing system and arXiv.

Each instance provides a model with an earlier sample report and a new collection of evidence documents. The model must generate an updated report. A reference report supplies the comparison target, but is not given to the model.

For each company or research topic, the researchers construct three chronological snapshots. These produce two update tasks: move from the first snapshot to the second, then from the second to the third. This design separates the old report’s organizing role from the newer documents’ evidential role.

Importantly, these are not simply untouched professional reports. Language models help transform the source material into evidence documents and synthesize the reference reports. Later reference outlines adapt earlier ones to preserve continuity. Financial filings are reorganized into coherent Markdown documents with tables; academic papers undergo a related conversion process. The benchmark therefore tests performance against a deliberately constructed reporting workflow.

The inputs are substantial. Average evidence collections range from about 24,000 to 59,000 tokens, the text units models process. Reference reports average roughly 9,000 to 23,000 tokens.

Three separate measures of success

FRAME evaluates structure, coverage and faithfulness separately.

For structure, a model extracts a report’s hierarchy into a tree containing elements such as sections, subsections and tables. An alignment algorithm compares generated and reference trees, allowing meaningfully similar elements to match even when their wording differs.

Coverage asks whether the output includes information from the reference report. An automated evaluator checks a sample of 100 extracted information units, awarding partial credit for incomplete matches. This measures recall against the constructed reference, not completeness against every potentially relevant fact in the original sources.

Faithfulness works in the other direction: it samples 100 checkable claims from the generated report and assesses whether the evidence supports them. Claims with weak initial support receive a deeper check for logical entailment. Because scores allow partial support, a faithfulness score is not simply the percentage of sentences that are true.

Similar organization, different substance

The researchers compare GPT-5, Claude-4.5-Sonnet, Qwen3-VL-30B-A3B, Phi-4-multimodal and Kimi-VL-A3B using the same one-shot generation prompt.

Qwen provides the clearest example of structural similarity without comparable content performance. Its overall structural score is 0.315, close to Claude’s 0.314 and above GPT-5’s 0.298. But its coverage score is 0.548, compared with 0.770 for Claude and 0.779 for GPT-5. Its faithfulness score is 0.545, versus 0.843 and 0.914 respectively. The other two open-source models also have lower structural scores, so the results do not show uniform parity across that group.

A Salesforce example makes the distinction concrete. All five outputs receive similar structural scores, yet the three open-source models label the reporting quarter incorrectly. Phi-4 also reports revenue of $9.44 billion rather than approximately $9.83 billion. Familiar headings do not prevent outdated or unsupported content.

The authors report stronger results on financial than academic reports. Their grouped analyses also associate longer reference reports and evidence collections with poorer structure and coverage. More table-rich reports tend to show better coverage and faithfulness, particularly for open-source models. These are observed associations across benchmark instances, not controlled demonstrations that adding tables causes improvement.

What remains uncertain

The evaluation itself needs scrutiny. GPT-5 performs structure extraction and judges coverage and faithfulness while also being a tested generator. The authors acknowledge potential model biases. Human validation covers just 10 financial instances, not the academic categories; agreement is stronger for coverage than for faithfulness.

The synthetic references and two source domains also limit generalization to clinical, legal or other professional writing. This is a preprint, not an established demonstration of deployment readiness. This account uses the supplied PDF text extraction; figure layouts and plotted details may not survive extraction reliably, so numerical comparisons here come from the reported tables and prose.

FRAME’s useful distinction is straightforward: preserving a document’s form does not establish that its update is complete or supported. Those properties require separate checks.