ConceptARC-100 tests how consistently reasoning models apply abstract rules

A larger test of rule learning

A model can produce a correct answer without reliably applying the rule a puzzle was designed to test. In the preprint ConceptARC at Scale: Studying Conceptual Understanding in Reasoning Models, posted on November 4, 2026, Philipp Mondorf and colleagues investigate this distinction using puzzles built from colored grids.

These puzzles follow the Abstraction and Reasoning Corpus, or ARC, format. Solvers receive several examples showing an input grid and its transformed output, then must infer the rule and produce outputs for new inputs. A rule might require recoloring objects inside a container or moving an object until it reaches a boundary.

The researchers ask whether models also represent relationships across different tasks. Two puzzles might require different transformations while both depending on the concept of “Inside and Outside.”

Building 1,600 reviewed tasks

The original ConceptARC benchmark contains ten handcrafted tasks for each of 16 concepts. The authors’ new generator combines three components: a grid layout, a selector identifying relevant objects or regions, and an action specifying their transformation. Concept-specific constraints restrict which combinations are allowed.

Using this generator, the team constructed ConceptARC-100: 100 new tasks per concept, totaling 1,600. Each task supplies three demonstration pairs and ten test inputs. Because a program defines each task, additional examples can be generated without manually drawing every grid.

The authors reviewed all tasks for consistent transformations and rules that could be inferred unambiguously from the demonstrations. An independent validation round flagged 6.2% as invalid; the authors report correcting these before further processing. They nevertheless acknowledge that human review can miss errors.

Correct answers do not guarantee consistency

The study evaluates eight open-weight reasoning models, including members of the Qwen, Gemma and GPT-OSS families, across three random seeds. Grids are presented as bracketed rows of numbers encoding colors. A prediction counts as correct only when its dimensions and every cell match the target.

Qwen3.5-27B achieved the highest individual-query accuracy, at 63.48%, followed by Gemma4-31B at 58.00% and GPT-OSS-120B at 45.41%. These figures describe the tested models and evaluation setup, not reasoning models generally.

Performance varied sharply by concept. Qwen3.5-27B scored 83.27% on “Center,” compared with 40.27% on “Order,” 36.27% on “Same and Different,” and 33.30% on “Count.” The authors suggest that the counting tasks’ output format—bars or stripes overlaid on the input—may contribute to difficulty, but do not establish that explanation.

The stricter task-level measure exposed a larger weakness. Qwen3.5-27B solved at least one test query in 93.21% of tasks, but all ten in only 23.17%. A relatively strong average therefore concealed frequent failures across inputs governed by the same rule.

More examples helped a smaller model

The researchers also trained two much smaller models from scratch on the demonstrations: VARC, with approximately 18 million parameters, and TRM, with approximately seven million. Each learned a task-specific vector used alongside new inputs from that same task.

With three demonstrations per task, VARC achieved 17.59% query accuracy and TRM 7.43%. Retraining VARC with 24 demonstrations per task raised its query accuracy to 77.4%, with all ten queries correct in 58.3% of tasks.

That result supports the authors’ conclusion that task-specific example availability is an important bottleneck. It is not an equal-information comparison with the reasoning models, which received only three demonstrations, nor a demonstration of transfer to entirely unseen tasks.

Internal structure favored operations

To examine internal representations, the researchers extracted model states after the third demonstration and averaged them across eight different demonstration sets per task. They then measured whether tasks sharing a layout, selector, action or concept lay closer together in representation space.

For Qwen3.5-27B, early layers primarily reflected grid layouts. Later layers showed stronger organization by actions. At layer 39, action-based separation reached 73.4%, compared with 64.5% for concept-based separation. These are representation-separation measures, not puzzle-solving accuracies.

VARC’s learned task vectors also grouped strongly by action. Visualizations showed clusters involving operations such as expanding object boundaries or applying checkerboard coloring, even across different concept categories.

Evidence of structure, not a causal explanation

The central limitation is that these representation findings are correlational. The study does not establish that models use the identified clusters to generate answers; interventions would be needed to test that relationship.

Models also performed better on the original, publicly available ConceptARC benchmark. The authors cannot rule out prior training exposure, but the difference does not prove contamination.

The preprint’s contribution is a larger, reviewed testing resource and evidence that operation-related internal structure can coexist with unreliable rule application. It does not establish human-like conceptual understanding or consistently generalizable reasoning.