DriveFoundry links driving-data corrections with selective AI reasoning

Making difficult experiences reusable

In the DriveFoundry preprint, Xiaomi researchers describe two connected improvement loops. Offline, verified failures guide corrections to training data and annotation tools. During driving, an inference scheduler decides whether a proposed maneuver needs more computation before selection.

The central data unit is a scene–candidate record. It links observations, a possible driving decision, its continuous trajectory and assessments of its consequences. Several candidates can share the same scene, allowing comparisons between alternatives rather than treating the recorded driver’s behavior as the only useful answer.

Annotations distinguish an object from a decision-relevant factor. A pedestrian’s presence is an object observation; that pedestrian entering the vehicle’s path is a factor that can justify yielding. Records connect such factors to decisions, including when each relationship applies and whether it motivates a maneuver or constrains its execution.

Rules, vision-language models and human reviewers contribute to these records. Observed outcomes remain separate from judgments about unexecuted alternatives, which can come from simulation, rules or expert review. That distinction matters: an imagined consequence is not a recorded event.

One model, several depths of computation

The model, Xiaomi-Drive-o1, uses a shared vision-language backbone for five tasks: perception, causation, action, world prediction and value assessment. Value evaluates candidates across safety, comfort, efficiency, compliance and navigation following.

Shared training lets auxiliary tasks influence the representations used for planning, even when those tasks are not executed during a particular decision. When additional tasks do run, a mechanism called latent refill passes compact internal representations between them. Future predictions and value assessments stay associated with the particular trajectory being considered.

Each decision begins with a relatively short action-and-value path. If its result is insufficient, a learned value-of-computation head estimates whether additional tasks or larger internal token budgets could improve the final choice enough to justify their cost. Expansions must fit remaining time and memory limits. Separate rule-based checks filter candidates; a predefined fallback applies when no acceptable candidate can be obtained within the budget.

The scheduler learns from offline experiments that expand computation from the same saved inference state. Its target is the estimated improvement in the selected candidate, minus added execution cost—not simply a preference for more reasoning.

What the benchmarks showed

The researchers evaluated recorded-trajectory prediction on PhysicalAI and interactive simulated driving in 1,607 AlpaSim scenarios.

For recorded trajectories, ADE@1 measures average displacement error for the selected prediction. The best-of-six measure, minADE@6, asks whether at least one generated candidate follows the recorded motion closely. Neither metric alone establishes safe interactive driving.

The nine-billion-parameter model achieved the lowest selected-trajectory error among the tested models at both evaluated horizons on both splits. On the curated held-out split, its five-second ADE@1 was 0.83 metres, compared with 1.51 metres for the 34-billion-parameter Alpamayo 2 Super. It did not lead every best-of-six comparison.

In AlpaSim, the nine-billion-parameter model had a 19.1% close-encounter rate, a 5.4% at-fault close-encounter rate and a 9.8% off-road rate. Alpamayo 2 Super recorded 21.9%, 5.7% and 10.3%, respectively. However, Xiaomi’s model achieved lower progress and lower AlpaSim scores. These results therefore show a trade-off, not an across-the-board victory.

A separate 0.8-billion-parameter ablation supports the computation-efficiency claim. Joint training reduced five-second best-of-six error from 1.93 to 0.89 metres—about 54%—without changing the fast inference path or token consumption. Adaptive inference used 28% of full reasoning’s tokens and produced broadly similar, but not identical, results: close encounters were 23.7% versus 23.5%, while five-second trajectory error was 0.80 versus 0.83 metres. Token consumption is not itself a measurement of energy savings or proportional speedup.

What remains uncertain

The standard PhysicalAI NV split overlaps publicly available training data by construction. The authors also report a separately curated held-out subset, but its challenging cases were manually selected.

The paper acknowledges that causal explanations agreeing with reviewed labels do not establish reliable causal behavior in unfamiliar situations. Annotation quality and scenario coverage remain limiting factors. World prediction receives direct visual supervision only through three seconds, although action plans extend to 6.4 seconds. Shared representations may also produce correlated prediction and evaluation errors, and scheduling estimates may become inaccurate after model or hardware changes.

Real-vehicle tests ran at 10 Hz on an NVIDIA RTX 4090, demonstrating runtime feasibility in the tested configuration—not comprehensive road-safety validation. The work is an arXiv preprint; its reported findings should be read as research results, not proof of deployment readiness.

Source: DriveFoundry, exact version 2611.11532v1.