AI coding assistants struggle between embedded software designs and working code

Beyond writing an isolated function

Can an AI coding assistant produce software that fits an existing embedded project, rather than merely looks plausible? RealEmbedBench, introduced in a November 2026 arXiv preprint by Yujun Huang and colleagues, examines that question through application-layer tasks: software logic built above the interfaces that connect programs to hardware.

This is an empirical evaluation of generated software, not a theoretical result or a demonstration of reliable operation on physical devices. Its central finding is that performance depends strongly on the task, the surrounding code and, for software migration, the direction of translation.

Building a benchmark from existing projects

The researchers used 125 developer discussion posts and interviews with 12 developers to help select three task categories: code generation and completion, state-machine development, and translation between operating systems. These consultations informed the benchmark; they are not a representative survey of the entire embedded industry.

The dataset draws on 15 open-source projects and contains 5,142 samples: 1,426 generation tasks, 3,026 completion tasks, 80 state-machine examples and 610 operating-system translation pairs.

Generation tasks ask models to reconstruct functions from descriptions and project context. Completion tasks hide tokens or larger code blocks. State-machine examples connect written requirements, PlantUML diagram descriptions and implementation code. Translation tasks span six operating-system families, including FreeRTOS, Zephyr and RT-Thread.

DeepSeek-V3.2 helped generate descriptions, state-machine representations and translation test inputs. The authors report manual review and correction. That makes this a curated, partly model-generated benchmark derived from existing code, rather than a collection consisting entirely of independently written developer requests.

Nearby code changes the answer

The team evaluated six models under an 8,192-token context limit. Gemini-3.1-Pro obtained the highest reported code-generation score, 0.7076, and completion score, 0.6915.

These figures should not be read as percentages of programs that work. Generation and block completion use CodeBLEU, which compares generated code with a reference through textual, syntactic and data-flow features. Token completion instead uses exact matching.

Removing context generally weakened generation scores. For Gemini-3.1-Pro, removing all context reduced CodeBLEU from 0.7076 to 0.4149. Removing only the nearby code window reduced it to 0.5929. Local code supplied useful constraints about control flow, helper functions and implementation patterns.

There are exceptions to the paper’s broad wording. Its table shows StarCoder scoring worse without the local code window than without any context, contradicting the claim that removing everything always caused the largest drop. The practical lesson is therefore narrower: context selection matters, and nearby code was particularly useful in these experiments.

A valid diagram is not a verified implementation

A state machine describes how a system moves between states when events occur or conditions hold. Recognizing those states is only part of implementing the required behavior.

Five models achieved reported PlantUML syntax-validity scores around 0.95–0.96. Yet coverage of reference states and transitions was lower, and the best state-machine code scores were 0.4177 from diagrams and 0.4379 from written requirements.

These measurements assess different things and cannot be compared as equivalent success rates. In particular, the implementation scores use CodeBLEU, not execution-based proof that the state machine behaves correctly. Nevertheless, inspection of 50 incorrect cases identified concrete problems: unsupported API calls, conditions checked at the wrong stage and simplified transition logic. Error categories could overlap.

Translation results need their denominators

Translation received stronger functional checking: generated code had to compile, execute and pass the supplied tests, with about three tests per pair on average. GLM-4.7 had the highest reported aggregate translation score, 0.7833.

The headline difficulty was Harmony-to-RT-Thread translation. Five models failed and one passed, producing an average of 0.1667 across models. But the dataset contains just one example in that direction. The reverse direction also contains one example, passed by every model. This is evidence about those particular cases, not a stable estimate of either migration’s general difficulty.

Limited tests cannot establish complete behavioral equivalence, especially for concurrency, timing or hardware interactions. Open-source selection also leaves proprietary systems underrepresented, and recent repository activity does not establish that training-data overlap was absent.

RealEmbedBench offers useful diagnostic tasks, not deployment certification. Its most actionable message is to provide relevant project context and check generated behavior explicitly—especially the conditions, interfaces and return values that plausible-looking code can get wrong.