Learning Multimodal Embeddings for Traffic Accident Prediction and Causal Estimation

We consider analyzing traffic accident patterns using both road network data and satellite images aligned to road graph nodes. Previous work for predicting accident occurrences relies primarily on road network structural features while overlooking physical and environmental information from the road surface and its surroundings. In this work, we construct a large multimodal dataset across six U.S. states, containing nine million traffic accident records from official sources, and one million high-resolution satellite images for each node of the road network. Additionally, every node is annotated with features such as the region’s weather statistics and road type (e.g., residential vs. motorway), and each edge is annotated with traffic volume information (i.e., Average Annual Daily Traffic). Utilizing this dataset, we conduct a comprehensive evaluation of multimodal learning methods that integrate both visual and network embeddings. Our findings show that integrating both data modalities improves prediction accuracy, achieving an average AUROC of $90.1%$, which is a $3.7%$ gain over graph neural network models that only utilize graph structures. With the improved embeddings, we conduct a causal analysis based on a matching estimator to estimate the key contributing factors influencing traffic accidents. We find that accident rates rise by $24%$ under higher precipitation, by $22%$ on higher-speed roads such as motorways, and by $29%$ due to seasonal patterns, after adjusting for other confounding factors. Ablation studies confirm that satellite imagery features are essential for achieving accurate prediction.

💡 Research Summary

The paper tackles the problem of traffic‑accident risk prediction and causal factor estimation by leveraging both road‑network structure and high‑resolution satellite imagery aligned to individual road‑graph nodes. The authors first construct a massive multimodal dataset covering six U.S. states (Delaware, Massachusetts, Maryland, Nevada, Montana, Iowa). The dataset contains over nine million recorded accidents, one million 1024×1024 satellite images (≈200 m × 200 m coverage per image), and a rich set of auxiliary attributes such as weather statistics (temperature, precipitation, wind speed, pressure), road type (residential, motorway, etc.), and edge‑level traffic volume (Average Annual Daily Traffic, AADT). Data collection required extensive preprocessing to harmonize heterogeneous DOT records, align schema, and geospatially match accidents and images to the OpenStreetMap‑derived directed road graph.

For modeling, the authors propose a multimodal learning framework that fuses graph neural network (GNN) embeddings with visual embeddings extracted by a Vision Transformer (ViT). The GNN processes node and edge features through standard message‑passing layers, while the ViT encodes each satellite image into a dense vector representing road geometry, lane count, surrounding land‑use, vegetation, and other visual cues. Three fusion strategies are evaluated: (1) naïve concatenation followed by an MLP, (2) cross‑attention between graph and image embeddings, and (3) a meta‑learning approach that learns modality‑specific weighting. The cross‑attention design yields the best performance, achieving an average AUROC of 90.1% across the six states. This represents a 3.7‑percentage‑point improvement over a strong GNN‑only baseline (AUROC ≈ 86.4%). Ablation studies show that removing image features drops AUROC by 3.5 points, confirming the essential role of visual information; removing weather, traffic‑volume, or graph features leads to smaller but still notable declines (1.8–3.7 points).

Having obtained high‑quality multimodal embeddings, the authors conduct a causal analysis using a matching estimator. They define treatment groups for three key factors: high precipitation (top quartile), high‑speed roads (motorways), and seasonal variation (winter vs. summer). For each treated node, a control node is selected as the nearest neighbor in the embedding space, thereby balancing observed confounders captured by the multimodal representation. The average treatment effect on the treated (ATT) is estimated as 24.2% increase in accident risk under heavy rain, 21.9% increase on motorways, and 28.6% increase during winter. These ATT estimates are corroborated with propensity‑score matching (PSM) and double‑robust (DR) estimators, indicating robustness to model specification.

The paper’s contributions are threefold: (1) Release of the largest publicly available multimodal traffic‑accident dataset, with precise alignment of satellite imagery to road‑graph nodes; (2) Demonstration that integrating visual embeddings with graph embeddings substantially improves accident‑risk prediction; (3) Application of the learned embeddings to causal inference, quantifying the impact of environmental and infrastructural factors while controlling for confounders.

Limitations include the static nature of the satellite images (no temporal updates to capture seasonal road‑surface changes), potential residual confounding beyond what the embeddings capture, and reliance on publicly available DOT data that may have reporting biases. Future work is suggested to incorporate time‑varying remote‑sensing data (e.g., Sentinel‑2, drone footage), vehicle‑on‑board sensor streams, and more sophisticated causal graph models to disentangle direct, indirect, and interaction effects among risk factors.