VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture

Traffic signal control (TSC) is a core challenge in urban mobility, where real-time decisions must balance efficiency and safety. Existing methods - ranging from rule-based heuristics to reinforcement learning (RL) - often struggle to generalize to complex, dynamic, and safety-critical scenarios. We introduce VLMLight, a novel TSC framework that integrates vision-language meta-control with dual-branch reasoning. At the core of VLMLight is the first image-based traffic simulator that enables multi-view visual perception at intersections, allowing policies to reason over rich cues such as vehicle type, motion, and spatial density. A large language model (LLM) serves as a safety-prioritized meta-controller, selecting between a fast RL policy for routine traffic and a structured reasoning branch for critical cases. In the latter, multiple LLM agents collaborate to assess traffic phases, prioritize emergency vehicles, and verify rule compliance. Experiments show that VLMLight reduces waiting times for emergency vehicles by up to 65% over RL-only systems, while preserving real-time performance in standard conditions with less than 1% degradation. VLMLight offers a scalable, interpretable, and safety-aware solution for next-generation traffic signal control.

💡 Research Summary

VLMLight introduces a hybrid traffic signal control (TSC) framework that unites vision‑language meta‑control with a dual‑branch reasoning architecture to address both efficiency and safety in urban intersections. The authors first develop a novel image‑based traffic simulator capable of providing multi‑view visual inputs (bird’s‑eye and directional cameras) in real time. These images are processed by a Vision‑Language Model (VLM) acting as a Scene Understanding Agent, which generates structured natural‑language summaries describing lane‑level vehicle types, congestion levels, and the presence of special vehicles such as ambulances.

The textual scene descriptions are fed to a Large Language Model (LLM) meta‑controller (Agent ModeSelector). This meta‑controller decides, based on semantic cues, whether the current situation is routine or safety‑critical. For routine traffic, VLMLight invokes a lightweight Transformer‑based reinforcement learning (RL) policy that consumes spatio‑temporal tensors of recent vehicle counts and occupancy across up to twelve movements. The RL branch operates with sub‑10 ms latency, ensuring real‑time responsiveness.

When the meta‑controller detects safety‑critical conditions (e.g., an emergency vehicle, a potential conflict, or an accident), it switches to a deliberative reasoning branch. In this branch, three specialized LLM agents—Phase Reasoning, Signal Planning, and Rule Verification—collaborate through a structured dialogue. They sequentially (1) analyze phase‑level traffic conditions, (2) select a green phase that gives priority to the emergency vehicle, and (3) verify that the chosen action complies with traffic regulations and system constraints. This collaborative process mimics human traffic‑control reasoning, providing interpretability and auditability while leveraging the generalization power of LLMs.

Extensive experiments across a variety of traffic densities, emergency‑vehicle arrival rates, and intersection geometries demonstrate that VLMLight reduces emergency‑vehicle waiting time by up to 65 % compared with RL‑only baselines. In standard traffic scenarios, the framework incurs less than a 1 % degradation in average delay and queue length, confirming that the fast RL branch maintains high performance. Ablation studies show that removing the VLM‑generated visual context reduces performance by over 20 %, highlighting the importance of visual grounding. The meta‑controller achieves over 98 % accuracy in correctly routing to the appropriate branch, indicating reliable detection of safety‑critical events.

Key contributions of the paper include: (1) the first image‑based traffic simulator that supplies multi‑view visual data for TSC, enabling rich perception beyond traditional vectorized state representations; (2) a safety‑prioritized LLM meta‑controller that dynamically selects between a low‑latency RL policy and a high‑level reasoning module; (3) a collaborative LLM reasoning pipeline that provides transparent, rule‑compliant decisions for critical situations; and (4) empirical evidence of substantial improvements in emergency‑vehicle handling while preserving routine efficiency.

Limitations are acknowledged: the simulator does not yet model complex environmental factors such as varying illumination, weather, or sensor noise; LLM inference latency, though acceptable in the deliberative branch, may still be a bottleneck for large‑scale deployment; and the threshold criteria used by the meta‑controller rely on expert‑defined heuristics. Future work will focus on integrating real‑world video streams, optimizing LLM inference with hardware accelerators, learning meta‑controller thresholds automatically, and exploring meta‑learning approaches that jointly train RL and LLM components for tighter integration.