A smarter split could help vehicle AI spend less energy

Connected vehicles may need to analyze camera and LiDAR data quickly, but running a large vision model entirely on an onboard computer can be demanding. One alternative is split inference: a vehicle runs part of a model, sends an intermediate representation over a wireless link, and lets a nearby edge node finish the job. The computers and the link both have limits, and those limits can vary with competing workloads and network conditions.

A fixed split point may not suit every situation. Sending data earlier in the model can reduce the work done onboard, but the intermediate data may be larger. Running more layers locally can reduce what must be transmitted, but increase the vehicle’s computing load. Lower numerical precision—using INT8 rather than FP16, for example—can reduce computation and data size, but may also reduce detection accuracy.

The authors propose an optimizer that considers these choices together. It uses a mixed-integer linear program, or MILP, to select a split point for each stream and a numerical precision for each model layer. Its objective is to minimize computing and vehicle-to-edge communication energy while meeting limits on completion time and task quality. Layer precision is shared across streams, while the split point can vary by stream. The optimizer uses measured device performance, link rates and layer-wise estimates of how quantization affects quality.

The team tested the approach using a 12-layer vision transformer on NVIDIA Jetson Orin Nano and Orin NX devices, each paired separately with an Orin AGX edge aggregator. Workloads were derived from the OPV2V dataset and included two, four, six or eight concurrent streams. Vehicle-to-edge link conditions were emulated, with rates selected from tested values. The researchers compared the method with fixed FP16 splits, running everything locally in FP16, and a fixed split using INT8 throughout.

Across the tested configurations, the proposed method reduced communication energy by 33.3–36.0% compared with the uniform INT8 split. This does not mean it always used substantially less total energy: at eight streams on the Orin Nano, total energy was 3.07 joules versus 3.16 joules for uniform INT8. The optimizer’s objective was energy subject to a latency requirement, not minimizing latency itself.

At eight streams, average task completion latency was 98.4 milliseconds on the Orin Nano, compared with 149.2 milliseconds for uniform INT8; on the Orin NX, it was 86.7 milliseconds versus 129.5 milliseconds. Detection accuracy was 75.1–76.1% mAP50 on the Nano and 75.3–76.1% on the NX, compared with 79.2% for the FP16 reference. The reported gap from FP16 was 3.1–4.1 percentage points. In these tests, the optimizer kept some layers in FP16 and used INT8 in others, reflecting measured differences in layer sensitivity.

These findings are bounded by the evaluation. It used specific Jetson hardware, one model and dataset-derived workloads, with emulated rather than live vehicle-to-edge links. The communication-energy estimate relies on a specified transmission-power value; actual energy and performance may differ with radios, network conditions, vehicle workloads and software. The quality constraint uses calibrated sensitivity estimates, while final detection accuracy is measured separately. The results do not establish performance or safety in real traffic, or show that the optimizer generalizes to other models and vehicle systems.

The source is an arXiv preprint, so these results should be read as a research report rather than as settled evidence. The study supports a narrower conclusion: on this testbed, jointly choosing precision and split location improved several reported measures relative to the tested uniform INT8 split, while retaining some accuracy relative to FP16. Real-world validation remains necessary.