Can language models map admissible next steps in math?

A fluent suggestion is not necessarily a structurally valid next step in a course. A model might recommend a task before its prerequisites, or name one admissible task while overlooking others. To assess this specific problem, researchers need a reference that defines which tasks are allowed from a given state of mastered knowledge.

Peter Steiner and Jan Hochweber propose a benchmarking method based on Knowledge Space Theory (KST), a framework for representing task prerequisites and the knowledge states they permit. Their sample study applies the method to small mathematics domains. It evaluates agreement with a specified structure, not whether recommendations improve learning.

Turning prerequisites into a test

Consider a curriculum in which determining a linear function’s equation from its graph involves identifying the slope and intercept. More generally, a learner’s mastered tasks form a “knowledge state.” Given a task catalogue and prerequisite relations, KST can derive admissible states and, for each state, every unmastered task that can be added next without violating the prerequisites. The authors call this complete set the “outer fringe.”

The benchmark asks a model to return all and only the tasks in that set. Researchers compare the model’s recommendations with the reference as unordered sets of task identifiers. Exact-set accuracy is strict: a response is correct only if it includes every admissible next step and no inadmissible ones. Other measures, including precision and recall, help distinguish extra recommendations from omitted options.

The sample study tested six language models in four domains involving linear functions. The domains, drawn from an existing Swiss educational project, contained 23 tasks and 50 knowledge states in total; every state was tested. Four prompt conditions supplied different amounts of structural information: task descriptions alone; a request for the model to infer prerequisites; a supplied prerequisite relation; or the complete list of admissible states. Researchers also varied the catalogue order and collected 2,400 responses.

Supplied structure, higher agreement

Exact-set accuracy was 61.3% with descriptions alone and 62.8% when models were asked to reconstruct prerequisites. The small difference did not provide clear evidence that asking models to infer the structure improved recommendations in this study.

Accuracy was higher when prompts supplied the reference structure: 96.3% with prerequisite relations and 98.2% with the complete knowledge space. These scores show that the tested models often reproduced the reference sets under those conditions. They do not show that either prompt format is better for teaching, or that a model is better than direct calculation. When the prerequisite relation and knowledge state are known, the admissible set can be computed deterministically.

Performance varied by model and domain, and no model consistently led across all domains and prompt conditions. Domain differences were smaller when the reference structure was supplied. The authors caution that an overall ranking across prompts may not guide selection for a particular application: results depend on the task and the information available to the model.

Across the two catalogue presentations, recommendation sets matched in 85.25% of paired comparisons. That agreement measure is not the same as correctness: two outputs can agree with each other and still differ from the reference. Nor can the study attribute every difference to ordering. The two presentations were not identical-prompt repeats, and the design cannot separate possible order effects from response variability.

What the scores leave open

The reference is a specified curriculum structure, not a universal account of the best learning sequence. A high score means agreement with that structure. It does not establish that the structure is appropriate for every classroom, identify which admissible option would be most helpful to an individual, or evaluate the quality of explanations. The study did not assess student experience or learning outcomes.

The empirical evidence is limited to four small linear-functions domains, the tested model versions, and the prompts and settings used. The authors say broader studies are needed to assess the method across larger structures, subjects, languages, and curricular settings. The study also did not include repeated calls with identical prompts, which would help distinguish stochastic variation from presentation effects.

This work is an arXiv preprint; the supplied source does not establish peer-review status. Its sample study illustrates a formal method for comparing recommendations against an explicit prerequisite-based reference. Whether that agreement is useful in practice—and whether it benefits learning—requires separate evaluation.