ARC-AGI-2 study finds better scheduling and memory use matter more than extra synthetic training

A reasoning test with a deadline

ARC-AGI-2 presents a few examples of transformations between coloured grids, then asks a solver to produce the exact output for a new input. Grids can contain up to 30 by 30 cells. A solver gets two attempts per test input, but the hidden evaluation runs without internet access on four NVIDIA L4 GPUs for at most twelve hours.

In their preprint, Rules, Traces and Routing Statistics, Nguyen Minh Thang and Trinh Thi Phuong report 189 evaluation runs. Their available hidden results cover 120 semi-private tasks; private-set scores were unavailable. Scores average each task’s fraction of correctly answered test inputs, rather than simply counting all inputs equally.

The central finding is specific to the systems tested: additional training data often helped less than changes that let models use their limited time and memory more effectively.

Synthetic puzzles passed checks, but did not improve transfer

The researchers constructed 12,000 executable rules and expanded them into 95,993 synthetic tasks. Rules covered operations such as selecting objects, combining shapes, repairing symmetry and tracing paths.

Admission required different behaviour signatures on a fixed probe bank. Each task also had to distinguish its rule from shortlisted competitors and related program variants, while defeating twelve simple shortcut solvers. An executable oracle checked the output labels.

These checks have boundaries. Distinct signatures do not prove that all rules compute distinct functions, and rejecting finite competitors does not establish that a task uniquely identifies its rule among every possible transformation. The authors did not test human inferability. Although the base rule release received an independent audit, the expanded task collection did not receive a new independent audit.

For the NVARC Qwen3-4B grid model, continued training on 1,920 real-task records raised the mean semi-private score from 28.96 to 31.72. Those records represented 1,147 task identifiers because official datasets overlap. Revised scheduling and context lengths subsequently produced a five-run mean of 33.06, with a best run of 34.86.

No synthetic-trained checkpoint tested alone beat the real-task model, and adding such checkpoints alongside it did not raise the mean. Training with the entire rule corpus yielded 28.61 in one run.

The generated puzzles differed substantially from ARC-AGI-2: inputs were generally larger, every input contained six colours and a palette key, and outputs were usually smaller. The authors suggest these differences may restrict transfer, but did not test that explanation. Their negative result does not establish that synthetic data generally cannot help.

Better inference, not a demonstrated training breakthrough

For Qwen3.8-27B, the team assembled 1,036 self-generated reasoning traces whose final grids matched known answers. Filtering removed forced completions, repetitive loops and other undesirable traces.

Fine-tuning alone did not produce a clear accuracy improvement at a matched reasoning budget. In one offline comparison, the fine-tuned model solved 63 of 172 inputs versus 57 for the base model; the difference was not statistically significant.

The larger system improvement came from inference changes. The serving engine enforced a thinking limit within a single request, avoiding expensive rereading of context. After answering, the model wrote a Python transformation that was checked against the demonstrations. Scheduling prioritised a long first attempt for every input before starting second attempts.

Together, the fine-tuned model and revised procedure reached 44.03, compared with an earlier base-system score of 26.81. That best run was near the ARC Prize Foundation’s reported 42.36 for the base model without a time limit. It does not establish superiority or equivalence: the protocols differed, and the small score difference was unresolved by single runs.

Removing experts freed room for more reasoning

The third model, Qwen3.8-Flash-Next, uses a mixture of experts: only part of its 125 billion parameters is active for each token, but storing its weights still consumes substantial memory.

Pruning from 512 to 384 experts per layer expanded the key-value cache, which stores information used during generation, from roughly 0.55 million to 1.39 million tokens. A later 296-expert configuration, adapted using ARC training-task routing and protected code-writing experts, reached about 1.96 million cache tokens with additional cache changes.

The main observed benefit was more simultaneous requests, not substantially faster individual decoding steps. With a program-checking agent and revised token allocation, the best run scored 56.67.

What the results establish—and leave open

These are preprint findings about complete systems, not isolated rankings of model intelligence. Several improvements changed multiple factors together. Public evaluation tasks appeared in training for affected models, while expert lists were selected using public evaluation results.

Repeated NVARC configurations showed about 1.94 points of run-to-run variation; repeats were insufficient to estimate variability reliably for the other models. The strongest supported lesson is therefore practical and bounded: under this hardware deadline, memory capacity, executable checks and token allocation were central to performance, while the tested synthetic training additions did not deliver the hoped-for gains.