Can an AI model’s familiarity with code reveal how students wrote it?

A student submits a Python program that works. Did they develop the solution themselves, adapt an online example, ask an AI assistant, or combine all three? Those routes can produce similar-looking submissions while involving different learning processes.

Subhankar Maity and colleagues investigate whether language models can supply clues about code origins. Their study, posted as an arXiv preprint on November 6, 2026, examines a technique called membership inference. It reports experiments on public code and a small student hackathon study. The findings are preliminary, and the central distinction matters: detecting possible training-data membership is not the same as detecting AI authorship.

Looking for familiarity, not a signature

Language models assign probabilities to tokens—the pieces of text or code they process—based on the preceding context. Membership inference uses those probabilities to estimate whether a sample appeared in a model’s training data.

The researchers adapt an existing method called Min-K%. Rather than averaging confidence across an entire snippet, it concentrates on the tokens the model finds least predictable. The underlying idea is that previously encountered code may contain fewer unexpectedly difficult pieces than unfamiliar code.

The system averages selected low-probability token scores and compares the result with a calibrated threshold. It requires access to token probabilities, not just an assistant’s conversational answer. However, familiarity alone does not establish a snippet’s history. A student might independently write a common programming pattern, retrieve an online example, or receive similar code from an AI assistant.

Building the comparison

For the public-code experiment, the researchers used Python answers from ProCQA, a dataset drawn from Stack Overflow. They treated these as member examples: code presumed to have been encountered during training. The evidence presented does not establish that each tested model actually trained on those particular answers.

They constructed supposed non-member examples by transforming the original code. Changes included renaming variables, altering structure, introducing redundant or broken logic, converting code into other languages, and generating more distant configurations or query patterns.

This creates an important limitation. The comparison may partly measure whether models distinguish ordinary Python answers from heavily altered or different-language material, rather than whether they recognize their actual training examples. Structural difference also does not establish absence from a training corpus.

The team allocated 30% of the data to threshold calibration and 70% to evaluation. Locally evaluated models, including GPT-2, DistilGPT-2 and DialoGPT variants, were tested on approximately 3,000 samples per class. Models accessed through paid APIs—including GPT-5, Gemini, Mistral, DeepSeek and LLaMA variants—were tested on approximately 200 per class because of budget limits.

A signal, but weak detection at low false-alarm rates

DistilGPT-2 achieved the highest reported area under the receiver operating characteristic curve, or AUC: 0.713. AUC summarizes how well a score separates the two labeled groups across thresholds; 0.5 corresponds to chance-level ranking. Gemini-2.5-Flash led the API-tested group with 0.654.

A more demanding measure asks how many positive examples are detected when false positives are limited to 5%. GPT-2 led the locally evaluated models at 18.19%, while GPT-5 led the API-tested group at 10.14%. Even under the study’s labels, most positive examples therefore went undetected at that operating point.

The authors suggest that privacy protections help explain lower detection scores among API-tested models. But these experiments do not isolate the effects of particular privacy mechanisms. Different models, access arrangements and sample sizes complicate that comparison; lower detection performance is not itself proof of stronger privacy.

What happened with student work?

The educational proof of concept involved five consenting groups, each comprising seven higher-education students. Each group submitted a Jupyter notebook for a hackathon task predicting passenger survival in the Titanic dataset. Students also voluntarily supplied their AI conversation logs.

The researchers extracted code and comments from those logs and used several exact and approximate matching methods to create reference annotations for AI-derived content. Against those annotations, AUC ranged from 0.584 to 0.644. At a 5% false-positive rate, the method detected only 6.25% to 8.24% of annotated positive segments. Precision also varied substantially between groups.

Qualitative inspection suggested that students frequently sought AI assistance with feature engineering, alongside imports, model training and evaluation plots. These observations concern five group submissions, not programming students generally.

The study’s useful question is how teachers might better understand the process behind finished code. Its results do not establish a dependable authorship detector or demonstrate improved learning outcomes. Any future educational use would need to preserve the distinction between a probabilistic clue, documented assistance and evidence about a student’s understanding.