Why an Edge AI Chip’s Speed Is Only Part of the Story

When an AI system on a camera or other edge device recognizes an object, the neural processing unit (NPU) is only one part of the job. Software may resize and normalize the image before inference, then decode the model’s output afterward. Those steps can add time—and changing them can also affect the prediction’s accuracy.

A technical report by Youngjin Jo and colleagues asks how to compare edge NPUs fairly when vendors recommend different model settings and software pipelines. The key distinction is between improving the model’s execution on the NPU and optimizing the entire route from image input to final prediction.

Three ways to measure performance

The researchers tested four commercial NPUs: Hailo-8, Hailo-10H, DeepX DX-M1 and Mobilint Regulus. They ran three vision models: ResNet50 for image classification, YOLOv5m for object detection and YOLOv8s-pose for pose estimation.

They compared three configurations. The Baseline uses the transformations needed to run a model on the NPU but limits optional vendor-specific optimizations. Vendor Guided NPU Only (VG-NPU) applies vendor-recommended model transformations while keeping the model’s inputs and outputs, host-side processing and benchmark runtime the same as the baseline. This offers a more controlled view of model transformation. Vendor Guided End-to-End (VG-E2E) follows the vendor’s recommended deployment pipeline, including its input format, preprocessing, runtime and postprocessing.

This distinction matters for timing. Runtime-call latency measures the runtime API invocation, including NPU execution and overhead inside that call. System-level latency covers the broader journey from loading an image to producing a prediction. The team evaluated accuracy on 5,000 images per workload, using the same evaluation images and calibration data across configurations. Latency tests used batch size one and synchronous, single-stream execution.

Faster does not always mean more accurate

In the controlled VG-NPU comparison, both accuracy and runtime-call latency improved in 10 of the 12 NPU–workload combinations. Runtime-call latency fell in 11 combinations overall, while accuracy improved in 10. These results suggest vendor-guided model transformations can sometimes make inference faster while maintaining or slightly improving measured accuracy—but the benefits were not universal.

The full vendor-native pipeline showed a different pattern. VG-E2E reduced system-level latency relative to the baseline in all 12 combinations, but accuracy improved in only four and declined in eight. In other words, a faster end-to-end deployment did not necessarily preserve the baseline’s accuracy.

The study also found that NPU-only rankings do not always tell the whole story. For ResNet50, the ranking of three NPUs tested on the same host changed when the researchers moved from runtime-call timing to end-to-end timing: DX-M1 was slowest in the former comparison but fastest in the latter. For the two YOLO workloads, the ranking stayed the same. The relationship between accelerator timing and full-system timing therefore depended on the workload.

Compared with VG-NPU, VG-E2E had lower system-level latency in every tested combination, with additional reductions ranging from 10.4% to 70.7%. But VG-E2E changes several elements at once. The researchers therefore cannot attribute those gains to any one factor, such as preprocessing, postprocessing or the runtime pipeline.

What the results can—and cannot—tell us

The practical lesson is not that one vendor or NPU is universally best. It is that an NPU benchmark and a deployment benchmark answer different questions. A team choosing hardware should measure the complete pipeline it intends to use, while tracking accuracy alongside latency. Chip-level speed alone may miss important differences in software and data handling.

The findings have important boundaries. The tests cover four NPUs, three vision workloads and specific vendor software stacks, with synchronous batch-one execution. Some platforms used different hosts, so their system-level timing cannot be attributed solely to the NPU. The study did not measure power or energy, and it does not establish whether lower latency means lower energy use. It also does not isolate the contribution of each change in the full vendor pipeline or explore every possible compiler and quantization setting. The authors caution that their results describe the tested configurations, not maximum achievable performance or universal superiority.

This is an arXiv technical report, a preprint rather than evidence of peer review. Its results are best treated as a useful, bounded benchmark study—not a definitive ranking of edge AI hardware. The report’s central contribution is methodological: compare controlled model changes and full deployment separately, because they can produce different trade-offs.