Keeping the right evidence when AI inputs must shrink

Two questions about shorter inputs

CoRA-Bench examines a practical problem in long-input AI: how to remove text without discarding information needed to answer a question. Its central distinction is between preserving evidence and using that evidence successfully. A selection method can perform well on one measure without leading on the other.

Huzaifa Pardawala, Siddhant Sukhani, Sudheer Chava and Agam Shah describe the benchmark in an arXiv preprint submitted on November 6, 2026. The findings concern the tested datasets and model configuration, rather than establishing a universally best selection method.

A shared budget, different selection strategies

The researchers compared keeping the beginning and end of a source; BM25, which ranks passages using lexical matching; dense retrieval, which uses learned representations to find relevant passages; and dense retrieval followed by reranking. They also tested LLMLingua-2, which removes tokens, and Provence, which ranks passages and prunes sentences.

The main comparisons imposed limits of 32,000, 64,000 and, where applicable, 128,000 tokens. Tokens are the units used to divide and count text for model processing. These limits covered selected source text, excluding the question and instructions. A common tokenizer provided consistent accounting, although its counts can differ from those used by the reader model.

Retrieval methods worked with overlapping 512-token chunks. Selected overlaps were merged, and retained passages were returned in their original order. Crucially, evidence measurements and answer evaluations used the same final text.

Four settings tested different demands: BABILong placed supporting facts within long book text; LOONG Financial Spotlight involved financial documents; a custom NoLiMa diagnostic tested questions with little literal overlap with the relevant passage; and LongMemEval supplied multi-session conversation histories.

What survived selection?

Evidence recall measures how much of the annotated supporting material remains after selection, averaged across eligible questions. The definition depended on the dataset: supporting facts, automatically identified financial candidate lines, an inserted target passage, or answer-bearing conversational turns.

BM25 led supporting-fact recall on BABILong at every tested budget, reaching 92.1% at 128,000 tokens. The ordering changed on the custom NoLiMa diagnostic: at 64,000 tokens, dense retrieval and dense retrieval with reranking each retained 90.8% of target passages, compared with 46.7% for BM25.

Dense retrieval also retained 99.3% of annotated answer-bearing turns on LongMemEval at 64,000 tokens. These results favor lexical matching when explicit word-level cues are available, and semantic retrieval in the tested settings with limited lexical overlap or conversational evidence.

Surviving evidence is not enough

The team then used Gemini 2.5 Flash-Lite to answer questions, repeating evaluation three times on the same questions and applying dataset-specific scoring rules.

At a 32,000-token budget, BM25 achieved 46.20% answer accuracy on BABILong, compared with 31.13% using the full source. Provence reached 48.13%, but its estimated advantage over BM25 was uncertain: the paired 95% confidence interval for their difference included zero. The comparisons were exploratory and were not adjusted for multiple testing.

On LongMemEval, dense retrieval achieved 71.13% accuracy at 32,000 tokens, versus 63.20% with full context. On the custom NoLiMa diagnostic, Provence reached 27.43%, compared with 0.57% for full context—even though dense methods led evidence recall at 64,000 tokens. These differing rankings show why both measures matter.

Important boundaries

Exact-text evidence scoring can penalize methods that alter wording while retaining useful meaning. For extractive methods, the benchmark instead traces evidence to its original source occurrence. LOONG’s candidate lines are automatically identified, not human-annotated gold evidence, and its eligible questions change with the budget, complicating comparisons across budgets.

The NoLiMa test is a custom repeated-measures diagnostic, not the official benchmark aggregate. LongMemEval tests selection from a fixed history, not a memory system updating through ongoing conversations.

Answer results also depend on one reader model with thinking disabled and a fixed output limit. String matching and automated judges can make mistakes. Shared token budgets do not establish equivalent model-native costs, and recorded selection-compute measurements are not end-to-end latency estimates.

The study’s practical conclusion is therefore conditional: evaluate what a shortened input preserves and what the reader can actually do with it, rather than assuming either measure alone identifies the best method.