Traffic-VL learns to recommend traffic-light phases from camera views

A queue of cars and a queue containing an ambulance can look identical once both have been reduced to a vehicle count. That information loss motivates Traffic-VL, a system described in a new arXiv preprint by Xingchen Zou and colleagues. Their eight-billion-parameter vision-language model recommends traffic-light phases directly from synchronized roadside images and written descriptions of the movements each phase permits.

The proposal is not unrestricted AI control of traffic lights. In its field deployment, the model supplies priorities to an existing signal controller, which retains responsibility for approved sequences, timing limits, pedestrian service and clearance intervals.

Learning from synthetic streets

Traffic-VL uses Qwen3-VL 8B as its backbone. At a standard four-approach junction, it receives four ordered 512-by-512-pixel images. Phase descriptions identify the relevant viewpoint, lane and movement, such as a left turn or straight-through movement. The model must connect those descriptions to visible demand rather than rely on an externally constructed table of lane counts. Different layouts can use different numbers of views and candidate phases.

The authors also introduce SynTra-V, a synthetic-data pipeline. SUMO supplies road layouts and vehicle states, Blender renders corresponding scenes, and gpt-image-2 changes their appearance to resemble roadside surveillance under varied lighting, weather and city styles. Structural reference images constrain vehicle positions, headings and road markings.

Because a convincing-looking image could still move a car into the wrong lane, four separate Gemini 3 Pro audit calls check vehicle layout, road geometry, depth ordering and movement direction. Every check must pass. This filtering is imperfect: calibration against three traffic experts on 500 candidate views found a 2.8% false-accept rate.

Training has two stages. First, reinforcement learning rewards correctly formatted answers that agree with expert-preferred phases, using approximately 5,000 photorealistic multi-view samples and annotations originally collected from 11 traffic experts. Both the visual encoder and language model are updated; experts do not supply detailed reasoning traces.

Second, the policy interacts with CityFlow traffic simulation, receiving rewards based on queue lengths and waiting times over multiple decisions. This stage uses deterministic Blender images for consistent, efficient rollouts. The visual encoder is frozen to preserve the surveillance-style adaptation learned earlier. Nearby intersections also exchange summaries of expected directional traffic.

What the comparisons show

Across five Jinan and Hangzhou simulation benchmarks, Traffic-VL closely matched Traffic-R1, the authors’ earlier controller supplied with privileged simulator-derived traffic descriptions. Tests over five paired demand seeds supported equivalence within a prespecified 2% margin for average travel and waiting times.

On Jinan1, Traffic-VL’s average travel time was 269.71 seconds, compared with 333.06 seconds for Gemini 3 Pro and 300.53 seconds for an adapted VLMLight pipeline receiving the same images. These results emphasize transfer without retraining on the evaluation networks. Conventional reinforcement-learning controllers became competitive when trained separately on each target benchmark, so the findings do not establish universal superiority over locally optimized control.

The simulation also paused while synthetic observations and decisions were produced. Its traffic metrics therefore do not include the real-time cost of image generation.

On Real-1296, an open-loop benchmark of 1,296 real multi-view samples from 14 intersections outside training and deployment, Traffic-VL selected an expert-approved phase in 82.1% of cases, versus 75.0% for Gemini 3 Pro. About 15% of samples allowed multiple acceptable phases. This measures agreement with expert judgments, not traffic-flow improvements from executing those choices.

Removing photorealistic stylization reduced real-image accuracy to 66.0%; allowing the visual encoder to update during simulator training reduced it to 73.4%. These comparisons support the authors’ separation of visual adaptation from policy refinement.

A constrained field trial, not causal proof

The team operated Traffic-VL for three weeks across ten interconnected intersections, evaluating 15 weekday evening peaks. Average queue length was 30.9 vehicles, compared with 34.5 under historical manual operation, a reported 10.4% reduction. The historical Traffic-R1 result was 31.3 vehicles.

Those comparisons used the same sites, measurement system and evaluation window, but the reference results came from an earlier study—not simultaneous randomized assignments. They demonstrate operation alongside favorable historical comparisons, rather than isolate how much improvement Traffic-VL itself caused.

The controller could extend a currently recommended green within its maximum duration or advance through the approved sequence after minimum green requirements were met. Median field decision latency was 6.3 seconds, with a 95th percentile of 9.2 seconds, against a nominal 15-second deadline.

Important limits remain. Each decision uses still images without explicit motion history. The model can miss emergency vehicles, hallucinate priorities or send misleading flow estimates to neighboring intersections. Certified signal constraints prevent infeasible transitions but cannot eliminate poor decisions based on misunderstood scenes. The preprint presents evidence for bounded camera-based assistance, not proof that unrestricted autonomous signal control is safe.