Language Models Can Read the Facts Yet Still Struggle to Justify an Answer

Testing reasoning after the evidence arrives

Retrieval-augmented generation, or RAG, supplies language models with external information to help answer questions. But a correct answer alone does not reveal whether a model followed that evidence or relied on information learned during training. When the two conflict, answering from memory can mean ignoring the supplied context.

The Rel-Bloom preprint, posted on November 4, 2026, examines this distinction by removing retrieval entirely. Researchers give models the relevant fact directly, then test their understanding of the relationship it expresses. This isolates one component that RAG depends on; it does not evaluate a complete RAG system.

One set of relations, several kinds of questions

Rel-Bloom contains 450 four-option multiple-choice questions built from 30 curated Wikidata relations. Each relation has one representative factual example, reused across five question types and three knowledge conditions.

The question types borrow names from Bloom’s revised taxonomy of human learning objectives. Remember asks for an explicitly stated value; Understand asks for a faithful paraphrase; Apply tests the same relationship in a new statement; Analyze requires distinguishing it from closely related but incorrect alternatives; and Evaluate asks for its strongest defining justification. Create, the taxonomy’s sixth level, is omitted because it requires open-ended generation.

The three conditions change entity information while preserving the relationship being tested. Factual uses the original true statement. Counterfactual replaces one entity with an incorrect real entity of the same broad type. Neutralized substitutes typed placeholders for names, reducing identity-based cues without introducing a conflicting real-world claim.

GPT-5 assisted with question construction. The authors manually reviewed questions and answers, then used Gemini and Grok to flag disagreements for further inspection and revision.

The evaluation covered Arabic, Chinese, English, Hindi, Persian and Spanish. One setting translated the entire item; another retained English facts and questions but translated the answer options. All 43 models received fully translated items, while only the 41 open-weight models received the hybrid setting. Together, the experiments produced 226,800 predictions.

Two different performance gaps

Every evaluated open-weight model series showed lower accuracy on Analyze than Apply in both settings, although the size and statistical support of the drop varied across model-size groups. In the fully translated setting, the Large Granite group dropped 47.2 percentage points, compared with 7.2 points for Large Qwen3.5. Neither Gemini model showed this Apply-to-Analyze decline.

Evaluate exposed a separate weakness. Gemini 3.1 Pro averaged approximately 99.5% across the other four question types but scored 60.7% on Evaluate. An Evaluate gap appeared in all eight open-weight series and both Gemini models, though statistical support varied by size group.

These results do not establish distinct human-like cognitive abilities. Evaluate’s incorrect options are deliberately plausible but non-defining, whereas Analyze’s alternatives can be rejected by checking the stated fact. Harder alternatives, rather than a different underlying reasoning operation, could therefore help explain the gap. Some defining criteria may also permit reasonable disagreement despite review.

Familiar names can help or interfere

Counterfactual statements were generally the hardest condition. However, the proposed ordering—Neutralized outperforming Factual, which outperforms Counterfactual—received statistical support in only a few model-size groups.

Removing names often helped relatively more on Analyze and Evaluate than on Remember and Understand, particularly among smaller open-weight models. That pattern weakened or reversed in several stronger models. It is not evidence that anonymizing information universally improves reasoning.

The authors also inspected one checkpoint, Qwen2.5-32B-Instruct, for cases where neutralization corrected a factual-condition error. Some wrong answers selected a familiar entity attribute instead of the requested relationship. These examples are consistent with competition from learned associations, but they do not establish an internal causal mechanism. The study did not directly measure whether each model had memorized each fact.

Shared difficulty, bounded conclusions

Items and relations that were difficult in one language tended to remain difficult in others. For fully translated items, average item-level rank correlations were about 0.54 for Gemini 3 Flash and 0.74 for Gemini 3.1 Pro. Shared English source items and translated templates may contribute to this agreement; it does not establish language-independent reasoning ability.

Larger model groups generally performed better, but differed in training and sometimes architecture, preventing causal conclusions about size alone. Whether Wikidata relation frequency had a negligible effect remained inconclusive; those counts do not measure training exposure.

The statistical analysis accounted for questions sharing the same source relation. Nevertheless, just 30 relations, each represented by one example, limit generalization. The benchmark also excludes noisy retrieval and open-ended answers. Its practical message is narrower than a verdict on RAG: evaluate different question operations separately, test conflicting and neutralized evidence, and measure accuracy directly in each language.