Testing whether language models understand Brazilian Portuguese on its own terms

What does it mean for a language model to understand Brazilian Portuguese?

Fluent answers can conceal specific gaps: a model might write smoothly yet struggle to conjugate a verb, identify how a word was formed, or interpret a colloquial expression. The authors of Brasilidade set out to make such skills more visible with a benchmark designed around Brazilian Portuguese rather than relying only on tests translated from other languages.

The name means “Brazilianness,” but the study does not measure national identity in any comprehensive sense. The authors describe a planned benchmark with four domains: language and grammar; Brazilian culture and general knowledge; tasks in Brazilian educational and institutional contexts; and safety and bias. This paper reports results only for the first domain, which covers language variation and grammar.

A set of tasks, not one general score

The researchers assembled 27 tasks using eight datasets, with more than 4,000 unique entries and almost 15,000 entries across the benchmark, according to the paper. The tasks include recognizing colloquial and regional expressions, classifying words, correcting grammatical errors, forming augmentatives and diminutives, conjugating verbs, and identifying word parts called morphemes.

The datasets were built in different ways. Some material was written or curated by the team; other material was collected from sources such as websites or generated with AI assistance. The authors say linguists reviewed AI-generated material, either through representative sampling or exhaustive inspection. They also say the datasets were designed to balance relevant fields where possible, for instance the regional distribution of expressions in certain tasks.

The team evaluated 19 open models, ranging in reported size from 2 billion to 70 billion parameters. They used the LM Evaluation Harness, a shared software framework, and standardized the prompts. In “generation” tasks, models received instructions and five or ten examples before answering. In “probability” tasks, models saw an incomplete sentence and the system selected whichever answer option the model assigned the highest probability. These are different testing conditions, not simply two ways of presenting an identical exam: the generation format provides examples and instructions, while the probability format does not.

Scores also depended on task type. The benchmark used exact-match scoring for some answers, accuracy or weighted F1 for classification, and BERTScore for open-ended answers. BERTScore estimates similarity between a response and a reference, so it is not the same as a human judgment that an answer is fully correct.

Examples helped, but some tasks remained difficult

Across the reported results, few-shot generation tasks—those with examples in the prompt—generally scored substantially higher than probability-based tasks, often by more than 10 percentage points. That pattern suggests examples and instructions can help models handle these tests. It does not show that prompting alone builds deeper knowledge, or that the same advantage would appear in other settings.

Model size was not a simple guide to performance. The authors describe a slight overall scaling pattern after excluding models with unusually poor outputs, while also reporting that some medium-sized models did especially well. Mistral-Small and Magistral-Small were above average in their categories, according to the paper. Portuguese-focused models Tucano and Sabiá performed around the average in this benchmark, despite results above average in some translated benchmarks. Because the study did not compare Brasilidade directly with those benchmarks, it cannot establish why the scores differ.

The toughest areas included verb conjugation, identifying types of word formation, segmenting words into morphemes, and identifying the region associated with an expression. Some models also produced empty or malformed answers: the authors report that two small Gemma checkpoints often returned empty responses in generation tasks, while GPT-OSS 20B frequently produced repeated question marks or other malformed output. They say the cause needs further investigation.

What the results can—and cannot—tell us

The main contribution is a reusable, task-specific way to probe aspects of Brazilian Portuguese that broad language tests can miss. The results also caution against treating either a model’s parameter count or its label as Portuguese-specific as a guarantee of performance on these tasks.

But the evidence has clear boundaries. Three of the four planned domains have no released tasks or results in this paper. The evaluation covers open models only, not major commercial systems named by the authors. The study does not compare its results directly with other benchmarks, and its few-shot examples were selected using a “take first” approach rather than deliberately varying labels. The authors also note that they did not use an AI judge for open-ended answers.

A benchmark is a sample of tasks, not a definitive test of a country’s language or identity. Regional expressions and grammar choices are only part of Brazilian Portuguese, and performance on these curated tasks does not by itself establish how a model will serve people in everyday use. Brasilidade’s remaining domains, and independent testing of its tasks and scoring choices, will be important to judging how broadly its conclusions apply.

The work is an arXiv preprint, so it should be treated as research that may not have undergone peer review. The paper links to a public project repository, but the findings summarized here are those reported by its authors.