Can exam-response models help tutors choose the next question?

The prediction problem

An adaptive tutor needs more than a list of questions: it needs a reasonable estimate of which question a learner is ready to tackle. Tom Quilter and colleagues ask a narrower, measurable question: given past responses and information about questions, can a model predict whether a student will answer an unseen question correctly?

The team used data from a large UK online mathematics revision platform. The data description reports responses from 49,521 students to 70 questions across three Edexcel GCSE mathematics mock papers, taken under formal, closed-book conditions and marked by qualified teachers. However, the paper’s results table lists 49,317 students for the three-paper evaluation. The source does not explain this difference, so the two counts should not be treated as interchangeable. For modelling, scores were converted into binary outcomes: a response counted as correct if the student earned more than half the marks available for that question. The authors describe the dataset as the largest open collection of professionally marked formal mathematics exam responses to date.

Comparing models

The researchers began with Item Response Theory, a family of models used in educational measurement. Their basic model estimates an ability level for each student and a difficulty level for each question. A student is predicted to be more likely to answer correctly when their estimated ability is higher or the question is easier.

They also tested richer models that allow performance to depend on topic-level strengths and weaknesses, such as algebra or geometry. One version estimates patterns for individual students; another estimates shared patterns for students in the same school class. The goal was to test whether that extra detail improves predictions.

The new method, discrete variational inference, represents uncertainty about a student’s estimated ability as a probability distribution rather than a single fixed number. It keeps the observed responses binary. The authors also tested active learning for newly enrolled students: instead of choosing questions at random, the system selects those expected to provide the most useful information for its predictions.

What they found

The basic ability-and-difficulty model reached about 81% accuracy in the reported full-data evaluations. Adding individual topic-level skill patterns did not improve predictions on the real exam data tested. A model with shared class-level topic patterns did perform somewhat better in the one-exam results reported. The authors suggest that class information may capture differences in what students have been taught. That is an interpretation of the model’s patterns, however, not proof that teaching caused the observed differences.

The variational method’s clearest advantage appeared when the researchers trained on smaller samples. At 15% of the one-exam dataset—about 5,300 students—the reported accuracy was 79.4%, compared with 78.7% for the standard class-interaction model. The authors report that the difference was statistically significant in their test. The gap was smaller or absent with more data, so this is best understood as a limited-data improvement, not a general leap in predictive performance. The authors also note that the result depended on careful hyperparameter tuning and a warm start from another model.

In an active-learning experiment involving 2,000 unseen students, choosing questions based on model uncertainty outperformed random selection after the first ten questions, according to the paper. The reported advantage eventually levelled at about one percentage point. This suggests a possible way to gather information efficiently, but the experiment measured prediction accuracy—not whether students learned more from the selected questions.

What the results do—and don’t—say

These are predictions of exam responses, not a trial of an AI tutor’s educational impact. Accuracy alone cannot show whether recommendations are motivating, fair or helpful for long-term learning. Nor does the failure of individual topic variables to improve prediction establish that students lack topic-specific strengths. The finding is limited to the models and exam data examined; the paper reports that its interaction model can recover topic structure in synthetic data where that structure was built in.

The dataset covers UK GCSE mathematics, and the models treat ability as stable during the exam window. Results may differ for other subjects, ages, regions, assessments or settings where skills change over time. The authors also identify greater training time and memory use as costs of variational inference compared with point-estimate alternatives.

The work is posted on arXiv. The evidence available here does not establish its peer-review or publication status, so no claim about that status is warranted. The paper presents a benchmark and a proposed low-data technique; whether these results transfer to real tutoring systems, or lead to better and more equitable education, remains an open question.