AI agents can build factory simulations—but running successfully does not mean being right

Building a virtual factory

In an arXiv preprint, the researchers present SimuLean, an open-source simulator built around the Unity game engine. It uses discrete-event simulation: instead of calculating every instant, the model advances between events such as a machine finishing a part, an operator starting a break or a vehicle completing a delivery.

An AI agent can assemble machines, queues, resource teams and transport routes through manufacturing-specific commands. These commands configure components whose behaviour has undergone platform testing, rather than requiring the agent to program every mechanism from scratch. The same configuration supports a 3D scene and runs without animation for repeated experiments.

Tested components reduce the amount of new software to inspect, but they do not guarantee that an agent has assembled the right factory.

How the models were judged

The benchmark covered 18 tasks, including four held-out tasks not used when designing the interface. Problems ranged from individual machines to production lines, shared operators and material handling. Two Claude models attempted tasks repeatedly, without human modelling guidance during the benchmark sessions.

An evaluation program ran the models left behind rather than trusting the agents’ reports. It compared their outputs with author-written reference models over 100 replications per model. Passing required statistical evidence of agreement for every target measure, within predefined margins: ±5% for throughput and utilisation, and ±15% for contents and waiting times.

Across two interface versions, the main agent produced valid models in 93 of 108 attempts, or 86%. On the benchmark interface version, its median time to a valid model was two minutes. Across both agents on that version, 105 of 108 attempts constructed models using domain commands alone. That measures traceability of construction, not correctness: a recorded sequence of commands can still describe a mistaken model.

The failures that looked successful

Across the main experimental block, 207 of 243 attempts were valid. Six stopped at a token limit; the other 30 failures were silent errors. They built and ran but produced answers that failed the reference comparison.

The most frequent mistake counted an operator as busy during a break because the agent paused the station without also stopping the operator. Other agents added an unspecified priority rule, misread time units or doubled demand that was already doubled in the starting model.

The build-and-run check caught none of these failures. Eight additional behavioural checks flagged 16 of the 30 silent errors, but also flagged 21 valid models. Those checks were written after the researchers knew the failure patterns, making their performance exploratory. An internally consistent simulation can still answer the wrong question.

A workshop comparison found an expert-model error

The industrial case reconstructed a published robotised welding workshop for components of offshore-wind foundations. It included shifts, overhead cranes, rail carts, shared workers and rework loops. Agents built models in both SimuLean and the Python simulation library SimPy.

Disagreement with the published FlexSim model prompted a stage-by-stage investigation. The researchers traced an error to stub-fitting operations reserving a crane from the wrong building span. After correcting the reference and aligning the specifications, all nine final agent-built models agreed with the corrected reference within 1%.

This was not independent confirmation against factory measurements. It was agreement between simulations. One current author also helped create the earlier model and prepared the reconstruction specification. The industrial SimuLean runs also required a fixed restart after reaching session limits, though no human intervened in their modelling decisions.

Faster code, more visible models

SimuLean was not the efficiency winner. On six benchmark tasks, SimPy models passed in all 18 attempts, versus 15 of 18 for SimuLean, and took less agent time and fewer tokens. In the industrial case, SimPy was also substantially faster and cheaper. The comparison was not perfectly matched: SimPy received explicit definitions of each performance measure and uniquely had shell access.

SimuLean instead offers an inspectable 3D model, reusable manufacturing components and integrated experiments. Its industrial studies ran inside the Unity editor, which was substantially slower than running the same engine outside it.

Separate transport tests showed that Unity physics could represent congestion missed by simpler travel assumptions, then supply calibration data for fast runs. However, Unity physics was itself a model—not measured vehicle behaviour—and the narrow-aisle physics experiments used only five runs per condition.

What remains uncertain

The evidence comes from author-designed tasks, two models from one provider, three repetitions per condition and one industrial case. Benchmark materials are available on request rather than openly bundled with the software.

The central result is promising but bounded: agents can construct substantial simulations, while successful execution and convincing animation remain insufficient evidence of correctness. Matching another model helps check implementation; establishing that either model represents a real factory still requires engineering validation.

Source and date

This article uses the supplied full text of arXiv:2611.10602v1, a preprint dated November 7, 2026. The manuscript supports that calendar date, but does not establish an exact submission time.