MIRAGE: Enabling Real-Time Automotive Mediated Reality

Traffic is inherently dangerous, with around 1.19 million fatalities annually. Automotive Mediated Reality (AMR) can enhance driving safety by overlaying critical information (e.g., outlines, icons, text) on key objects to improve awareness, altering objects’ appearance to simplify traffic situations, and diminishing their appearance to minimize distractions. However, real-world AMR evaluation remains limited due to technical challenges. To fill this sim-to-real gap, we present MIRAGE, an open-source tool that enables real-time AMR in real vehicles. MIRAGE implements 15 effects across the AMR spectrum of augmented, diminished, and modified reality using state-of-the-art computational models for object detection and segmentation, depth estimation, and inpainting. In an on-road expert user study (N=9) of MIRAGE, participants enjoyed the AMR experience while pointing out technical limitations and identifying use cases for AMR. We discuss these results in relation to prior work and outline implications for AMR ethics and interaction design.

💡 Research Summary

The paper introduces MIRAGE (Mediation Interface for Real‑time Automotive Graphics and Effects), an open‑source Unity‑based platform that brings Automotive Mediated Reality (AMR) into real vehicles. AMR is defined as a design space that combines Augmented Reality (AR), Diminished Reality (DR), and Modified Reality (ModR) to alter a driver’s or passenger’s visual perception before it reaches the eyes, thereby influencing Situation Awareness (SA) and downstream appraisal (trust, perceived safety, enjoyment).

Technical contributions: MIRAGE integrates state‑of‑the‑art computer‑vision models directly into Unity via the Unity Inference Engine, eliminating the need for external servers. It uses YOLO‑11 for semantic segmentation, DepthAnythingV2 for zero‑shot depth estimation, and MI‑GAN for generative inpainting. The pipeline runs at up to 34 FPS on consumer‑grade hardware, processing a front‑mounted RGB camera feed and projecting the result onto a head‑mounted display (HMD) with video pass‑through, effectively turning the HMD into a virtual windshield display.

The system implements 15 AMR effects grouped into three categories:

1. AR effects – object outlines, bounding boxes, icons, text labels, and other visual augmentations that highlight critical information.
2. DR effects – opacity reduction, blurring, transparent rendering, outline‑only display, and full removal of objects via inpainting, aiming to reduce visual clutter.
3. ModR effects – replacement of objects with alternative graphics, spatial transformations (translation, scaling, rotation), color changes, and artistic style transfers that reshape the perceived environment.

A dedicated UI panel allows expert users to enable, disable, and combine effects at runtime, and the architecture is modular so new models or effects can be added as plugins. The current prototype is intended for passenger‑seat use only; the HMD is not safe for manual driving.

Evaluation: An on‑road expert study with nine participants (four industry, five academia) was conducted. Each participant drove a vehicle for 15 minutes while wearing the HMD and configuring MIRAGE effects. Qualitative feedback indicated that participants found the visualizations engaging and helpful for awareness, but they also reported technical limitations: inpainting artifacts, depth estimation errors, and occasional mis‑classification leading to inappropriate removal or transparency of objects. Participants stressed the need for context‑aware gating of DR effects to avoid hiding safety‑critical cues.

The authors situate their work within a cognitive model of perception‑action loops: AMR interventions occur before visual perception, shaping SA (fast, moment‑to‑moment) and later appraisal (trust, safety perception). They argue that the role of AMR varies across SAE automation levels—AR is most beneficial for manual‑driving (levels 0‑2) to highlight hazards, while DR and ModR may support passengers in highly automated vehicles (levels 3‑5) by reducing overload and improving comfort.

Ethical considerations are discussed, including privacy concerns, the risk of over‑augmentation, and the importance of transparency (users must know what has been altered). Design guidelines such as user control, situational relevance, and minimal latency are proposed.

In summary, MIRAGE fills a gap in the literature by providing the first open‑source, real‑time framework that unifies AR, DR, and ModR for on‑road automotive contexts. It demonstrates technical feasibility, offers a rich set of visual effects, and provides empirical insights from expert users. The work paves the way for broader real‑world AMR research, encouraging the community to explore safety, usability, and ethical dimensions of mediated reality in vehicles.