Can a Small Selector Learn Which Examples Help a Language Model?

Choosing useful examples, not just similar ones

Large language models can perform tasks by reading labeled examples included in a prompt, without changing their own parameters. This is called in-context learning. Which examples they receive can influence whether they classify a question, review or news article correctly.

In the research preprint Can Jev Improve Demonstration Retrieval for In-Context Learning?, Rui Song, Yingji Li and Hao Xu investigate whether a specialized decision model can improve that selection. Their results suggest benefits across several classification benchmarks, but do not establish that the approach works universally.

A common selection method retrieves examples whose text embeddings—numerical representations of meaning—are closest to the query. The researchers instead ask a second question: would this labeled example help classify this particular query? Similarity still supplies the shortlist; a learned judgment changes its order.

A teacher judges; a smaller student learns

The initial approach retrieves 20 candidates using a sentence encoder. Jev then evaluates each candidate, considering the task description, query, example text and example label. Its probability that the example is useful becomes a ranking score.

That score is an estimate, not a measurement of how much the example actually improves an answer. Nevertheless, direct reranking improved average accuracy in several tested settings. With eight examples, JEV-9B raised average accuracy over similarity-based retrieval by 3.34 percentage points for Llama3.2-1B and 4.45 points for Qwen3-1.7B. The smaller NeoJEV-4B produced gains of 3.30 and 6.29 points, respectively.

The drawback is repeated computation: every shortlisted candidate needs a decision-model evaluation. The proposed solution, DisJEV, transfers those judgments into a lightweight multilayer perceptron, or MLP. A corresponding variant, DisNeoJEV, learns from NeoJEV.

The student uses frozen embeddings of the query, candidate and label, alongside features describing their relationship. It learns both to approximate the teacher’s scores and to preserve its preferences between pairs of examples. This combines learning how useful an example appears with learning which of two examples the teacher prefers.

Selecting a group of demonstrations

For distillation, the researchers sample 15% of each dataset’s training set, capped at 2,000 queries, and obtain teacher scores for 32 candidates per query. These queries are kept separate from test examples. A shared student is trained across the five datasets.

At inference, the student scores candidates without calling Jev. It also guides beam search: instead of simply taking the four highest-ranked examples, the system explores alternative combinations, retaining four promising partial sequences at each step. It represents each sequence through weighted combinations of example and label embeddings, giving greater weight to examples closer to the query in the prompt.

The main experiments cover question classification, binary and five-level sentiment classification, news topics, and natural language inference. They use four demonstrations and four open-source language models, with up to 2,000 test examples per dataset.

Higher averages, uneven benefits

DisJEV’s reported average accuracy gains over similarity-based retrieval were 2.99, 3.68, 0.43 and 1.68 percentage points across the four open-source models. DisNeoJEV achieved higher averages than DisJEV on all four. However, neither method improved every individual model–dataset combination.

Removing either training objective reduced average performance on the two smaller models examined in the ablation study. Training without the target dataset also retained some benefit: the held-out-dataset variant exceeded similarity retrieval by 1.56 and 1.98 points on those models. This supports transfer among the tested benchmarks, rather than unrestricted generalization to unfamiliar tasks.

Additional API experiments with two larger models reported smaller DisJEV gains, of 0.96 and 0.60 percentage points. Those comparisons used 500 samples and omitted the ConE and DiSP baselines because of confidence-estimation difficulty and cost.

What the evidence leaves open

The efficiency result needs careful interpretation. Against DiSP, reported selector-training time averaged 137.2 versus 497.7 seconds per 100 samples. Full in-context inference time was much closer: 68.6 versus 69.8 seconds. These measurements do not demonstrate a similarly large end-to-end inference speedup.

The paper reports no confidence intervals or repeated-run variability, leaving uncertainty around smaller differences. Its scope is also limited to classification benchmarks, not open-ended generation. Teacher–student pairwise agreement was approximately 70%, which indicates imperfect imitation and does not establish calibrated probabilities of usefulness.

There is a reporting inconsistency: the main table and appendix give different four-example Llama3.1-8B similarity-baseline averages, 78.47% and 74.80%. That complicates comparison across sections.

The preprint’s central finding is therefore promising but bounded: learning judgments about example usefulness can improve average classification performance while avoiding repeated Jev calls during deployment. Broader reliability and practical savings remain less firmly established.