An AI grader agreed on exam totals—but struggled to explain drawings

A coding agent takes up marking

In a preprint posted to arXiv on November 9, 2026, Rayed AlGhamdi of King Abdulaziz University examined Claude Code as a grading assistant for a course he teaches. The work is not established peer-reviewed evidence.

Sixty consenting students had their handwritten, 20-mark computer graphics midterms graded using Claude Opus 4.6 through Claude Code. Answers mixed Arabic and English. The exam included multiple-choice questions, short explanations, analytical and evaluative responses, and drawings showing the outputs of OpenGL graphics primitives.

The system received three files specifying grading rules, the answer key and the required output format, alongside reference images for the drawing question. After an initial sample paper, an orchestrator distributed the remaining papers among ten parallel sub-agents. The reported grading time was approximately 45 minutes—not a measurement of the complete instructor-supervised workflow.

The reliability analyses used smaller, separate groups: 20 papers for human–AI agreement and feedback assessment, and 15 different papers for a repeat grading session one week later.

Close totals, with an important qualification

For the 20-paper agreement sample, the reported correlation between instructor and AI totals was 0.97. Their mean absolute difference was 0.85 marks out of 20, and the AI awarded an average of 0.68 marks more.

Correlation describes how closely scores move together; it does not mean that 97% of marks were correct. A grader can rank students similarly while consistently awarding more credit.

There is a more consequential qualification: these were post-reconciliation statistics, not an untouched comparison of independent first-pass grades. The instructor initially graded blind to AI output, but subsequently examined disagreements. The paper reports seven AI grade overrides, six upward revisions to human grades and one retained human grade following review. Its results use the reconciled scores.

That process can improve students’ final grades. It also means the headline agreement cannot be read as the performance of an autonomous grader against an unchanged human benchmark.

On the separate 15-paper sample, the two AI runs achieved an intraclass correlation coefficient of 0.96, a measure of agreement across repeated scoring. Their mean absolute difference was 0.78 marks, with a maximum difference of two marks. Neither run’s agreement with the instructor was as close: mean absolute differences were 1.45 and 1.63 marks. Repeating a judgement consistently is not the same as getting it right.

Drawings expose the weak spot

The drawing task asked students to turn supplied vertices into line segments, a closed outline and a triangle. Grading required checking both connections and positions against reference images.

Drawing totals matched the instructor exactly on 55% of the agreement-sample papers. Across the two AI runs, only five of 15 drawing scores matched exactly. Drawing test–retest reliability had a point estimate of 0.52, with a wide confidence interval of 0.01–0.80: substantial uncertainty around an already weaker result.

Feedback made the limitation especially visible. Across all assessed items, the instructor coded 92.8% of explanations of lost marks as accurate, and 97.8% of confirmations of full-credit answers as accurate. For drawings, those figures fell to 27.3% and 55.6%, respectively, based on just 11 diagnostic and nine confirmation items.

Drawings were not worst on every statistic. Their weighted kappa, which measures score-category agreement while accounting for chance and disagreement size, was 0.74—within the same cluster as the other non-multiple-choice formats. Different measures revealed different weaknesses.

What went wrong—and what remains unknown

The author identified recurring failures: rewarding relevant vocabulary without sufficient explanation, misreading crossed-out multiple-choice selections, and allowing annotations or salient shapes to distract from drawing errors. In one example, the AI credited a drawing despite an incorrect vertex connection; in others, it missed incorrect coordinate placement.

The study is limited to one course, one institution and one model configuration. The instructor was also the sole human grading reference and feedback coder. There was no controlled comparison establishing whether the agent architecture itself improved performance, and the study did not establish learning benefits or assessment validity.

Students had instructor review, a review window before finalisation and an unconditional right to manual regrading. Those protections are part of the tested setting, not optional details. The evidence supports investigating supervised assistance—not replacing instructor judgement with a reassuring exam total.