NaijaFacts finds a gap between speaking Nigerian languages and recalling Nigerian statistics

Separating language from factual recall

A model that answers fluently in Yorùbá does not necessarily recall Nigeria’s fertility rates or census procedures accurately. Conversely, a wrong answer in Yorùbá does not, by itself, establish whether the difficulty lies with the language or the requested fact.

In the NaijaFacts preprint, submitted on November 4, 2026, Zainab Afolabi, Nikita Kozodoi, Jack Butler and Babs Khalidson investigate that distinction. Their benchmark varies both the language of a question and the country it concerns. The authors report that, on their statistical questions, the country-related accuracy gap is substantially larger than the language-related gap.

One question set, fifteen combinations

The benchmark starts with 100 questions about demographic and health statistics, including population counts, fertility and survey methodology. Native speakers and statistics experts from Obafemi Awolowo University authored questions in Nigerian languages; the team then produced English versions. Each question is available in English, Nigerian Pidgin, Yorùbá, Igbo and Hausa.

The researchers also constructed versions concerning the UK and US, using country-specific equivalents where a direct substitution would be unnatural. Crossing three countries with five languages creates 1,500 evaluated items per model.

Reference answers come from official sources. Nigerian answers draw primarily on the National Population Commission, National Bureau of Statistics and Nigeria Demographic and Health Survey. For the UK and US, a deep-research assistant retrieved answers from official sources, which the question authors then verified and corrected.

This design compares countries while keeping the query language fixed, and languages while keeping the requested country fact fixed. The authors call these contrasts the “knowledge effect” and “language effect.” These describe patterns in answer accuracy, rather than direct observations of what is stored inside a model.

The larger gap remains in English

Across eight evaluated models, average English-question accuracy was 20.1% for Nigeria, compared with 49.2% for the UK and 50.2% for the US. That is roughly a 30-percentage-point country gap without switching to a Nigerian language.

Language still mattered. On Nigerian questions, average accuracy fell from 20.1% in English to 17.5% in Pidgin, 15.1% in Hausa, 13.2% in Igbo and 11.2% in Yorùbá. The English-to-Yorùbá decline was about nine percentage points—smaller than the English-language country gap. On UK and US questions, that language decline was larger, at approximately 22–24 points.

The strongest systems also struggled with Nigerian statistics: their English scores reached 66–79% on the UK and US sets, but only 28–29% on Nigeria’s. Across all five languages, no evaluated model exceeded 31% on the Nigerian set. Two Nigeria-adapted, eight-billion-parameter models performed more poorly overall; their language adaptation did not eliminate the factual gap in this evaluation.

Switching the answer language barely helps

The researchers tested whether models might know an answer but struggle to express it in the requested language. They kept Nigerian-language questions unchanged while requesting English answers.

Average accuracy changes were small: +0.5 percentage points for Pidgin, −0.9 for Hausa, −0.2 for Igbo and +1.9 for Yorùbá. The authors interpret this as evidence against answer-expression difficulty being the main bottleneck. It does not establish that every failed fact was absent from training: the question still had to be understood, and the experiment measured responses rather than internal knowledge.

A wider test finds a domain divide

An English-only extension covered all 54 African countries, with 6,689 questions evaluated using Claude Opus 5. Statistical values were collected programmatically from World Bank and DHS Program interfaces, with indicators tied to specified sources.

The reported accuracy split was pronounced: 95.8% on culture, 97.7% on politics and 26.0% on statistics. These findings concern one model and this question collection, not every model’s knowledge of Africa. They nevertheless show why strong performance on capitals, leaders or landmarks need not imply reliable recall of quantitative indicators.

What the evidence leaves open

The authors report important limitations. The Nigerian set contains only 100 questions, and 81 draw on one survey report. Questions therefore cluster around shared tables rather than representing 100 fully independent pieces of evidence. The benchmark probes statistical knowledge, not broad knowledge about Nigeria.

Scoring also relies on one language-model judge, Claude Opus 5, without a human assessment or second judge. The continental extension uses only English and one answering model. Because the underlying sources are public, the authors cannot exclude their presence in training data.

The authors hypothesize that statistics buried in PDF tables are less readily absorbed than narrative facts. They propose retrieval over authoritative tables and better inclusion of statistical records in training. Those remedies were not tested here. The demonstrated result is narrower: in this benchmark, changing language alone did little to close a substantial gap in factual accuracy.