Similar examples give AI coding models a modest boost in a selected test

A reference shelf for programming puzzles

Competitive programming requires more than producing code that looks plausible. A solution must implement the right algorithm—a procedure for solving the problem—and pass tests that can expose incorrect logic or inefficient approaches.

In Assessing the Impact of Retrieval-Augmented Generation on Algorithmic Problem-Solving, Gabriel Souza Baggio and colleagues investigate whether language models benefit from seeing related, previously solved problems. The study is an arXiv preprint, version 2611.09791v1, submitted on November 6, 2026. Its findings are preliminary; the supplied materials do not document peer review.

The technique is called retrieval-augmented generation, or RAG. Instead of relying solely on information stored during model training, a system searches an external collection and supplies selected material alongside a new request. Here, that material consists of programming problems and their verified solutions.

How the comparison worked

The researchers used 13,328 problems from the CodeContests training split as their reference collection. They evaluated models on 81 of the dataset’s 165 test problems: only those for which their search system found at least one sufficiently similar example.

That distinction matters. The reference collection’s “training split” is a dataset partition, not a statement about what the language models themselves encountered during pretraining. The evaluated problems were excluded from the reference collection, but that alone does not rule out earlier model exposure.

To search the collection, the system converted problem descriptions into embeddings, numerical representations used to compare textual similarity. It retrieved up to three examples, using an adaptive similarity cutoff intended to limit irrelevant material. Matching used descriptions alone; the examples subsequently supplied to the model also included verified code, tags and other information.

Six models were tested: DeepSeek-R1, Gemini 2.0 Flash, Claude 3.7 Sonnet, GPT-4o, Qwen2.5-Coder-32B and GPT-3.5 Turbo. Each received problems under three conditions: the original problem without worked examples, the problem with randomly selected examples, or the problem with similarity-selected examples.

Generated programs were submitted to the CodeForces judging system, which checks solutions against hidden tests. The main measure, pass@1, was the proportion solved correctly on the first attempt.

Better matches, modest average gains

Averaged across the six models, similarity-based retrieval achieved a 20.37% success rate, compared with 17.9% without examples and 17.28% with random examples. The improvement over the no-example baseline was 2.47 percentage points, not 2.47 times the original performance.

DeepSeek-R1 performed best, rising from 61.7% without examples to 64.2% with relevant examples, while random retrieval reduced its result to 59.3%. On an 81-problem evaluation, the reported improvement corresponds to two additional accepted solutions.

The researchers also estimated Elo ratings, a difficulty-aware performance measure derived from which problems a model solved or failed. Average estimated ratings increased from 925 without examples to 1014 with similarity-based retrieval. These are statistical estimates from this benchmark, not ratings earned through actual contest participation.

Benefits were not universal across problem categories. For DeepSeek-R1, the estimated rating increased markedly on number-theory problems but decreased on data-structure and combinatorics problems. Relevant-looking examples can therefore still accompany worse results. The random-example comparison is consistent with the idea that additional context is not automatically useful, although it does not establish the precise reason for each failure.

What the study cannot settle

The evaluation deliberately selected problems with retrievable neighbors. It therefore measures performance where the system could find a match, not across the full test set or arbitrary new programming tasks. The paper does not report confidence intervals or repeated-run significance tests for its headline gains, leaving their statistical reliability uncertain.

A separate membership inference analysis, called Min-K, examined token probabilities for signs that solution code might resemble model pretraining material. The authors report that most analyzed code lines had membership confidence above 0.5, but this is indirect evidence, not proof that particular solutions were memorized.

There is also a reporting inconsistency: the paper says familiar material should receive higher token probabilities, yet its displayed decision rule labels higher average log-probability scores as non-members. That mismatch further complicates interpretation.

The clearest result is narrower than a demonstration of new reasoning ability: on this selected benchmark, supplying similar solved examples improved average judged correctness. How broadly that benefit transfers—and how much reflects adaptation rather than prior familiarity—remains unresolved.