Teaching Coding Agents the Hidden Structure of On-Device AI

Getting a new AI model to run on a phone is a software-engineering problem with many moving parts. An inference engine—the software that prepares and runs a model—may need updates to its model converter, computation graph, operators, and hardware-specific code. A change that looks complete in one part of the codebase can still fail when the model is compiled or run on an actual device.

The authors of EngineAgent argue that general-purpose coding agents often struggle because they may follow code-level connections without grasping the inference engine’s larger structure. They can change one component but miss a dependent one, or fail to trace a device-side error back to an earlier build or conversion step.

A map of the engine, plus prepared workflows

Before tackling a new model, EngineAgent builds a structured index that groups engine code by its role in the model’s journey: from input handling and graph construction to operators and hardware kernels. The index also records connections between layers, helping an agent plan changes beyond files that seem locally relevant.

The system prepares build, test, and debugging workflows for each hardware backend using models the engine already supports. When a new model arrives, a builder agent uses the index and workflows to make changes. An auditor checks the implementation layer by layer using the prepared checks. If problems arise, a debugger uses test results and logs to guide further fixes. The process includes end-to-end validation on the target device. The implementation is accepted only after it compiles, runs on-device, and produces output values—specifically, model logits—that agree with a reference implementation within the evaluation’s numerical tolerance.

That final check matters: a program can compile and run without producing correct results. Comparing its outputs with a reference asks for more than a successful installation or a plausible-looking response.

More completed tasks in a small benchmark

The researchers tested 12 model-enablement tasks involving three mobile language models and four engine–backend combinations. These covered CPU, GPU, and neural-processing-unit execution paths: MNN with a CPU, llama.cpp with Vulkan, MLC-LLM with OpenCL, and ExecuTorch with Qualcomm’s QNN backend.

EngineAgent completed 9 of the 12 tasks, a 75% success rate, with either of the two underlying language models used in the study. The strongest baseline, Codex, completed 4 of 12, or 33.3%. The paper also reports lower average task time and model-API cost than for baselines using the same underlying models. These results apply to this benchmark; they do not show that EngineAgent will outperform coding agents on every engineering task.

The authors’ component-removal tests suggest that both major additions helped in this evaluation. Without the hierarchy-aware index, success fell from 9 tasks to 3; without the prepared hardware workflows, it fell to 5. These tests do not establish that the same gains would occur in other settings.

The benchmark highlights the gap between making changes on a host computer and getting them to work on a phone. Baselines often made partial progress but failed at cross-layer implementation or hardware validation. In one case study, the authors describe EngineAgent using targeted tests and device-side debugging while adding a new attention operation to a Vulkan path. Such examples illustrate the system’s approach but do not independently confirm the benchmark results.

What remains uncertain

The evaluation covers 12 tasks, three models, and four engine–backend pairs. It cannot establish performance across the full range of phones, inference engines, model architectures, or development teams. Although the paper describes updating the index and workflows as engines change, the reported task results do not demonstrate reliable maintenance over long periods or under substantially different conditions.

EngineAgent also depends on its underlying language models and on the quality of its generated index, tests, and debugging procedures. Different models, prompts, or hardware could change the outcome. Agreement with reference logits verifies a defined form of numerical correctness, not every aspect of a model’s usefulness or real-world performance.

This work is presented as an arXiv preprint, and should be read as a reported demonstration on a focused benchmark. The findings suggest that explicit knowledge of software layers and device workflows may help agents with a difficult deployment task. They do not show that model integration on phones is solved or establish how the approach will perform more broadly.