Teaching GUI Agents to Assemble—and Repair—3D Machines

The difference between a valid click and a correct assembly

A graphical user interface (GUI) agent can perform an action that the software accepts while still producing the wrong assembly. It may choose the wrong component, misunderstand how parts relate, or move more pieces than a repair requires. That gap matters: successful interface control is not the same as sound mechanical reasoning.

In a preprint, Jingxuan Wei and colleagues present MAGAR-Bench, a benchmark for evaluating AI on virtual 3D mechanical assembly, and MAGAR, a system that selectively revises an existing model’s predictions. Their central idea is to treat assembly as three related but distinct problems: choosing an assembly operation and its parts; carrying out one native GUI action with the right parameters; and identifying the smallest set of parts that need to move to recover from an error.

What the benchmark measures

MAGAR-Bench contains 4,000 multimodal samples drawn from 26 assembly families. The samples combine visual information with details such as the assembly state, candidate parts, GUI context and interaction history. The authors divide the data by family: 20 families, with 3,000 samples, are used for training; six held-out families, with 1,000 samples, are used for testing. This tests performance on families not seen in training, though it does not by itself show how a system would perform across all kinds of CAD software or mechanical work.

The three tasks probe different abilities. Assembly Decision asks what operation to perform, which parts are involved and what the next subgoal should be. Atomic Action asks for a single interface operation—such as a click, drag, keystroke or text entry—and its parameters. Minimal Rework asks which parts must move to repair an incorrect assembly.

For that last task, “minimal” has a specific, limited meaning: the benchmark counts distinct moved parts among validated recovery candidates for a given state, not the number of GUI actions. The paper explicitly does not claim that this candidate set contains the globally best repair among every possible action sequence. Nor does matching a recovery route or action identifier prove that the repair would execute successfully; the benchmark does not run those recovery plans to verify them.

A cautious correction layer

MAGAR starts with a frozen, supervised fine-tuned vision-language model. Rather than always replacing its output, MAGAR generates possible revisions and uses task-specific selectors to decide whether a candidate should replace the original prediction. For assembly decisions and repair scope, it can revise structured fields while preserving others. For interface actions, it selects a complete action together with its parameters, instead of choosing those pieces independently.

That distinction reflects the different ways predictions can go wrong: changing one assembly field may leave other correct fields intact, while an action type and its parameters need to fit together. But a selective gate cannot guarantee that a proposed correction is right. Its purpose is to retain the original prediction unless a candidate meets the system’s learned intervention criteria.

Gains, with errors still in the ledger

On the held-out recorded states, MAGAR’s task-balanced Overall score was 41.39, compared with 37.50 for the supervised fine-tuned model it builds on—a gain of 3.89 points. The authors report gains across all three task families. This aggregate averages metrics within each family and then gives each family equal weight; it is not a measure of real-world assembly success or a simple percentage of tasks completed correctly.

The intervention accounting makes the tradeoff clearer. Among 209 cases where MAGAR changed the original prediction, 70 changed an incorrect prediction to a correct one, while 22 changed a correct prediction to an incorrect one. In 117 cases, strict case-level correctness did not change, though partial-credit metrics could still differ. These counts describe interventions, not the Overall score, and show why a correction system needs to be evaluated for both fixes and newly introduced errors.

What remains uncertain

The study evaluates recorded Unity states rather than long-horizon, live interaction in which an agent must act, observe consequences and adapt over time. The authors identify online assembly, globally consistent recovery planning and broader CAD environments as future directions. The benchmark’s repair target is also bounded by its validated candidate set, and its recovery identifiers are not execution-validated. So the findings support the value of task-specific correction in this test setting; they do not establish that MAGAR can reliably assemble or repair physical machinery.

Finally, this is an arXiv preprint, not evidence of peer-reviewed publication. Its reported results should be read as the authors’ evaluation under their stated data splits and metrics, pending further scrutiny and testing in other settings.