AI feedback lifts data-science project scores—but the judge matters

A complete project, not a single answer

In a November 3, 2026 arXiv preprint, Rohan Alexander examines whether language models can produce complete data-science projects and improve them through automated feedback. The reported gains are substantial on average, but the study also shows why higher AI-assigned scores should not be mistaken for independently verified scientific quality. The supplied source identifies a preprint, not a peer-reviewed publication.

The assignment was adapted from a third-year undergraduate statistics course. Models had to obtain data, clean and test it, conduct an analysis and write a paper using Quarto. Unlike a benchmark with one reference answer, the task allowed writers to choose their research question and dataset.

Seven models served as both writers and graders: Anthropic’s Claude Opus 5.5 and Haiku 4.5, OpenAI’s GPT-6 Sol and GPT-6 Luna, DeepSeek’s V4.1 Flash, Google’s Gemini 3.8 Flash and Z.AI’s GLM-5.3 Flash. Every writer–grader combination was run three times, producing 147 independent runs.

What the feedback loop changed

Writers initially received up to 40 minutes and 60 tool calls. Their submissions were then executed in a clean container, where scripts ran and the paper was rendered. A grader received the source files and execution report and assessed 26 rubric items. Writers could revise twice, each time with another 30 minutes and 40 tool calls in a fresh session.

Alexander reports that 107 of the 147 initial submissions ran end to end and rendered successfully. That rose to 116 after the final revision. Average scores from the assigned graders increased from 59% to 76%, with most of the improvement arriving after the first revision.

The averages conceal large differences. Opus 5.5 began at 92% and finished at 95%, leaving little room for improvement. Haiku 4.5 began at 7% and gained 46 percentage points. Some gains reflected avoiding rubric violations that triggered an overall zero, rather than incremental improvements throughout an analysis.

Revision was not uniformly beneficial. Sixteen runs finished below their initial score. Across 294 revision opportunities, 26 turned a working submission into one that no longer executed. In 24 of those cases, the writer exhausted its time or tool allowance before ending the session itself.

More work versus better advice

The study also tested revision without external feedback and single sessions with the full combined allowance of 100 minutes and 140 tool calls. These comparisons matter because a feedback loop also gives the writer more resources.

When scored by the six models other than the writer, final submissions averaged 73% with feedback, 65% after revision without feedback and 63% with the whole allowance upfront. These figures use a different scoring comparison from the headline 59%-to-76% result.

For four writers, one longer session came within three percentage points of the feedback condition. Haiku 4.5 behaved differently: it scored 45% with feedback, 8% after revision without feedback and zero in the longer-session condition. Alexander attributes much of that difference to graders flagging generated data. The results suggest that extra work explains much of the gain for some models, while explicit correction matters more for others.

Scores are not scientific validation

Repeated grading exposed measurement uncertainty. The same grader assessing an unchanged final submission produced scores differing by almost four percentage points on average. About 51% of third-round score changes were smaller than the grader’s own measured variation. This does not establish that those revisions accomplished nothing; it limits how confidently small score changes can be interpreted.

Self-grading produced scores about nine points higher on average than grading by other models, but the pattern was not universal. Alexander notes that these differences also reflect general grading strictness, so they are not a clean measure of self-preference alone.

A small fault-injection exercise provides a sharper warning. Alexander inserted seven separate faults into copies of one high-scoring project, including a false headline finding and a causal claim unsupported by an associational model. Only generated data substantially reduced the average mark. That single-project exercise is limited, but demonstrates that serious errors can coexist with high scores.

The main experiment did not establish analytical correctness through human expert review. Graders saw source text and execution records, not rendered figures. Writers also knew the detailed rubric, so improved marks may partly reflect better rubric compliance. The evidence supports gains in this particular workflow—not a general demonstration of reliable autonomous science.