Robustness Is a Function, Not a Number: A Factorized Comprehensive Study of OOD Robustness in Vision-Based Driving

Out of distribution (OOD) robustness in autonomous driving is often reduced to a single number, hiding what breaks a policy. We decompose environments along five axes: scene (rural/urban), season, weather, time (day/night), and agent mix; and measure performance under controlled $k$-factor perturbations ($k \in {0,1,2,3}$). Using closed loop control in VISTA, we benchmark FC, CNN, and ViT policies, train compact ViT heads on frozen foundation-model (FM) features, and vary ID support in scale, diversity, and temporal context. (1) ViT policies are markedly more OOD-robust than comparably sized CNN/FC, and FM features yield state-of-the-art success at a latency cost. (2) Naive temporal inputs (multi-frame) do not beat the best single-frame baseline. (3) The largest single factor drops are rural $\rightarrow$ urban and day $\rightarrow$ night ($\sim 31%$ each); actor swaps $\sim 10%$, moderate rain $\sim 7%$; season shifts can be drastic, and combining a time flip with other changes further degrades performance. (4) FM-feature policies stay above $85%$ under three simultaneous changes; non-FM single-frame policies take a large first-shift hit, and all no-FM models fall below $50%$ by three changes. (5) Interactions are non-additive: some pairings partially offset, whereas season-time combinations are especially harmful. (6) Training on winter/snow is most robust to single-factor shifts, while a rural+summer baseline gives the best overall OOD performance. (7) Scaling traces/views improves robustness ($+11.8$ points from $5$ to $14$ traces), yet targeted exposure to hard conditions can substitute for scale. (8) Using multiple ID environments broadens coverage and strengthens weak cases (urban OOD $60.6% \rightarrow 70.1%$) with a small ID drop; single-ID preserves peak performance but in a narrow domain. These results yield actionable design rules for OOD-robust driving policies.

💡 Research Summary

The paper tackles a fundamental problem in autonomous driving: how to evaluate and improve the out‑of‑distribution (OOD) robustness of vision‑based control policies. Rather than collapsing OOD performance into a single aggregate number, the authors propose a factorized evaluation framework that explicitly decomposes the driving environment into five semantically meaningful axes—scene (rural vs. urban), season (spring, summer, fall, winter), weather (dry, rain, snow), time of day (day vs. night), and agent mix (vehicles, pedestrians, animals). By defining “k‑factor OOD” test sets that differ from the in‑distribution (ID) training support on exactly k of these axes (k ∈ {0,1,2,3}), they obtain a robustness surface that shows performance as a function of both the number and identity of shifted factors.

The experimental platform is VISTA, a photorealistic, data‑driven simulator that supports closed‑loop roll‑outs. The authors train three families of end‑to‑end policies under identical data budgets and training schedules: a shallow fully‑connected (FC) network, a conventional convolutional neural network (CNN), and a Vision Transformer (ViT). In addition, they evaluate the impact of large‑scale frozen foundation‑model (FM) features by extracting patch‑wise descriptors from three pre‑trained encoders—DINO, CLIP, and BLIP‑2—and feeding them into a compact ViT policy head while keeping the feature extractor frozen. This isolates the contribution of generic visual representations learned from massive internet data.

Key findings are as follows:

1. Architectural hierarchy – ViT consistently outperforms CNN and FC across all OOD shells. The transformer’s global self‑attention appears to provide better invariance to the visual changes induced by the five factors.

2. Foundation‑model benefits – Policies that use frozen FM features achieve higher OOD success rates (often >85% even under three simultaneous factor changes) compared with training from scratch. The trade‑off is increased inference latency (≈30‑40 ms), which may be acceptable for many autonomous‑driving stacks but must be weighed against real‑time constraints.

3. Temporal context – Adding short frame histories (multi‑frame ViT‑Temporal or R‑CNN‑Temporal) does not surpass the best single‑frame ViT baseline. The modest temporal windows (a few frames) do not provide enough additional information to compensate for severe visual shifts such as night or heavy rain.

4. Factor impact – The most damaging single‑factor shifts are rural→urban and day→night, each causing roughly a 31 percentage‑point drop in route‑completion. Agent‑mix swaps incur ~10 pp loss, moderate rain ~7 pp, while season changes can be highly variable. Crucially, certain pairings (e.g., night + snow) are super‑additive, leading to larger drops than the sum of their parts, whereas other combinations partially offset each other.

5. Data‑curriculum effects – Training on winter/snow conditions yields the most robust single‑factor performance, whereas a baseline that mixes rural scenes with summer conditions gives the best overall OOD profile across multiple factor changes. Scaling the number of training traces from 5 to 14 improves average OOD accuracy by about 11.8 pp, demonstrating that sheer data volume helps but can be substituted by targeted exposure to hard conditions.

6. Diversity vs. specialization – Using a multi‑ID training set (covering several scenes, seasons, times) broadens coverage and lifts weak OOD cases (e.g., urban OOD improves from 60.6 % to 70.1 %). However, this comes with a modest drop in ID performance, indicating a classic trade‑off between specialization (peak performance on a narrow domain) and generalization (robustness across domains).

7. Interaction non‑additivity – The authors quantify interaction effects by comparing observed performance on k‑factor OOD shells with the sum of individual factor drops. Many interactions are non‑additive, confirming that a simple linear model of factor contributions would be misleading.

Overall, the paper delivers a rigorous methodology for diagnosing OOD weaknesses, provides empirical evidence that ViT backbones and frozen foundation‑model features are the most effective architectural choices for robust driving, and offers concrete design recommendations: (i) adopt ViT + FM features when latency budgets permit, (ii) construct training curricula that deliberately include night, snow, and urban scenes, (iii) prioritize data diversity when the goal is broad generalization, and (iv) recognize that short temporal histories alone are insufficient for OOD mitigation.

These insights are directly actionable for researchers and engineers building real‑world autonomous‑driving systems, guiding data collection, model selection, and system‑level trade‑offs to achieve safer deployment under the inevitable distribution shifts encountered on public roads.