Can AI agents improve how they work, not just their answers?

An AI agent’s abilities depend on more than its underlying language model. Prompts, tools, memory and the software controlling its actions determine how it approaches a problem. Could an agent improve that machinery while also improving its solution?

Flex-RSI, a preprint submitted on November 6, introduces a benchmark for that question. Its central distinction is important: the evaluated language model’s weights—the numerical parameters learned during training—never change. What changes is the software around it, called its harness, together with the program or workflow being developed. This is not evidence of a model autonomously retraining itself or achieving unlimited improvement.

Two jobs in an improvement loop

The benchmark gives the same model two alternating roles. An executor modifies the task solution, runs experiments and examines development data whose answers are available. An improver studies the execution records and revises the harness: its prompts, tools, workflow and accumulated lessons. It can also revise its own instructions, making the process recursive.

Candidate solutions face a separate validation set whose contents the agent cannot inspect. The starting acceptance rule requires an improvement larger than a margin based on measured starter-score variability; where variability was not measured, any increase qualifies. The improver can revise that rule, but cannot alter the evaluator, data splits or budget accounting. Hidden test data assess the resulting solutions after the run.

Every run begins with a shared general-purpose harness and a task-specific starter solution. Budgets constrain time, computing resources, model usage and evaluations, although the appendix lists model-specific exceptions.

Building systems leaves more room to improve

The suite contains 24 tasks spanning perception, reasoning, robot action, reconstruction, visual creation and computing infrastructure. Eighteen models appear in the experiments, but not every model attempts every task. The main analysis covers 20 tasks evaluated with at least six models; four others have only two-model comparisons.

The authors report that validation scores increased in 150 of 177 runs, or 85%, with a median gain of 10.4 points on the reported 0–100 scales. These are gains on validation data used during improvement, not an equivalent claim about hidden-test gains.

The largest opportunities generally arose when agents had to build a functioning system from a weak starter. In 3D object tracking, the median validation score rose from 19.4 to 62.2. The starter used color clustering and simple matching; agents could instead assemble systems using publicly available models and libraries.

By contrast, some question-answering starters already performed well. Median validation gains were zero on both ARC-AGI-1 and ARC-AGI-2. Low starting performance did not guarantee progress either: interactive ARC-AGI-3 games and reconstruction from video showed little median validation improvement.

Partial credit and physical checks

Feedback was another dividing line. Among tasks requiring agents to build solutions, open-weight models improved more when evaluators rewarded degrees of progress than when episodes yielded only success or failure. The median of task-level median gains was 9.9 points with graded feedback, versus 1.6 with success-or-failure feedback.

The proposed explanation is straightforward: partial credit can distinguish a useful intermediate change from a useless one before the entire task succeeds. However, these comparisons involve different tasks, rather than a controlled experiment changing only the feedback format. They support a pattern, not an isolated causal measurement.

Strict checks also exposed differences concealed by plausible appearances. On reconstruction from video, all four evaluated open-weight models received a physics component score of zero, compared with 70.7 for Claude Opus 5.5. That component requires dynamic objects to be reconstructed adequately and remain stable in simulation. Zero therefore does not establish that every generated object toppled over; missing or inadequately recovered objects also fail to earn credit.

Does rewriting the harness itself help?

Seven paired comparisons tested runs with and without harness editing. The editing-enabled run achieved a higher hidden-test score in six pairs and tied in one. For Qwen3.8-Flash-Next on Active Search, which rewards finding targets efficiently, the score increased from 21.0 with a frozen harness to 59.3 with editing.

That is useful evidence, but the sample is small. Runs generally were not repeated systematically, and the paper notes that another editing-enabled Opus run on the tied faucet task scored 40 rather than 80. Uneven model coverage, budget exceptions, small evaluation splits and automated judges on creative tasks further limit broad rankings. Extracted PDF text also does not reliably preserve figure imagery, so these comparisons rely on reported prose and tables rather than independent visual assessment.

Flex-RSI’s contribution is a way to test bounded improvement of an agent’s working process. Its results suggest that harness revision can help, while showing why repeated trials, informative feedback and checks beyond visual plausibility remain essential.