Spatiotemporal Semantic V2X Framework for Cooperative Collision Prediction

Intelligent Transportation Systems (ITS) demand real-time collision prediction to ensure road safety and reduce accident severity. Conventional approaches rely on transmitting raw video or high-dimensional sensory data from roadside units (RSUs) to vehicles, which is impractical under vehicular communication bandwidth and latency constraints. In this work, we propose a semantic V2X framework in which RSU-mounted cameras generate spatiotemporal semantic embeddings of future frames using the Video Joint Embedding Predictive Architecture (V-JEPA). To evaluate the system, we construct a digital twin of an urban traffic environment enabling the generation of d verse traffic scenarios with both safe and collision events. These embeddings of the future frame, extracted from V-JEPA, capture task-relevant traffic dynamics and are transmitted via V2X links to vehicles, where a lightweight attentive probe and classifier decode them to predict imminent collisions. By transmitting only semantic embeddings instead of raw frames, the proposed system significantly reduces communication overhead while maintaining predictive accuracy. Experimental results demonstrate that the framework with an appropriate processing method achieves a 10% F1-score improvement for collision prediction while reducing transmission requirements by four orders of magnitude compared to raw video. This validates the potential of semantic V2X communication to enable cooperative, real-time collision prediction in ITS.

💡 Research Summary

The paper addresses the critical need for real‑time collision prediction in Intelligent Transportation Systems (ITS) while respecting the strict bandwidth and latency constraints of vehicular V2X communications. Conventional solutions that stream raw video or high‑dimensional sensor data from roadside units (RSUs) to vehicles are infeasible because they quickly exhaust the limited wireless capacity and introduce unacceptable delays. To overcome these limitations, the authors propose a predictive semantic V2X framework that transmits only compact, task‑relevant embeddings of future video frames rather than the raw pixel data.

The framework consists of three main components: (1) a digital‑twin‑based dataset generation pipeline, (2) a semantic encoder built on the Video Joint Embedding Predictive Architecture (V‑JEPA), and (3) a lightweight decoder residing on the vehicle. Using the Quanser Interactive Labs (QLabs) digital twin, the authors synthesize 500 urban traffic video clips covering four intersection types (four‑way, three‑way, side roads, roundabouts). Each clip contains both safe‑driving and collision scenarios. After collection, the videos are post‑processed with YOLOv11 to detect vehicles and then transformed by one of three methods—heatmaps, binary road masks, or a hybrid of both—to emphasize traffic‑relevant regions and suppress background clutter.

V‑JEPA serves as the RSU‑side semantic encoder. It operates in a self‑supervised masked‑modeling fashion: video clips are split into non‑overlapping spatiotemporal patches, a “context” encoder processes the masked version while a “target” encoder processes the full clip, and a predictor learns to reconstruct the target embeddings from the masked tokens using an L1 loss. This training forces the model to learn predictive spatiotemporal representations directly in the embedding space, bypassing pixel‑level reconstruction. Once pretrained, the encoder parameters are frozen at deployment.

For downstream collision prediction, the frozen encoder outputs a sequence of token embeddings (size N × P × D). An “attentive probe”—a single‑query cross‑attention module—aggregates this sequence into a single 1 × D vector that captures the most salient motion patterns indicative of an imminent crash. The probe’s query vector is learnable and attends to the most informative spatial‑temporal locations. The resulting compact embedding is then fed to a lightweight linear classifier (binary output: collision vs. safe). Because only the 1 × D vector is transmitted over the V2X link, the communication payload shrinks dramatically.

The authors analytically compare payload sizes: raw video requires S_raw = N·H_o·W_o·3 bytes, whereas the semantic message needs S_sem = D·b bytes (b = 2 for FP16). In their experiments (e.g., N = 30 frames, H_o = 720, W_o = 1280, D = 512), the compression ratio R reaches 10⁴–10⁵, satisfying latency budgets (< 10 ms) for safety‑critical applications.

Experimental results show that the proposed framework achieves 92 % overall accuracy and improves the F1‑score by 8–10 % compared with baseline methods that transmit raw video or use only descriptive embeddings. The hybrid post‑processing (heatmap + binary mask) yields the best performance, confirming that emphasizing both dynamic objects and road geometry enhances the quality of the learned embeddings. Moreover, the transmission payload is reduced by five orders of magnitude, demonstrating the feasibility of bandwidth‑efficient, real‑time collision prediction.

Complexity analysis reveals that the frozen V‑JEPA encoder incurs O(L²·D) FLOPs (L = number of spatiotemporal tokens) during inference, but because its parameters are fixed, memory usage is dominated by O(L·D) token storage. The attentive probe and classifier together require only O(D² + D·C) operations (C = 2 classes), making them suitable for execution on typical vehicular embedded platforms.

The paper’s contributions are: (1) creation of a high‑quality, digitally‑twin‑generated video dataset with labeled collision events, (2) systematic evaluation of three post‑processing strategies to boost task‑relevant features, (3) adaptation of V‑JEPA for future‑frame embedding prediction in a V2X context, and (4) demonstration of massive communication savings while improving predictive performance.

Limitations include reliance on a simulated environment (potential domain gap to real‑world lighting, weather, and sensor noise), the need for substantial offline pre‑training of V‑JEPA, and the assumption of error‑free V2X links. Future work is suggested in the directions of multi‑RSU collaborative inference, robustness to channel impairments, model quantization/pruning for even smaller embeddings, and validation on real‑world vehicular testbeds.