Flow-Aware Diffusion for Real-Time VR Restoration: Enhancing Spatiotemporal Coherence and Efficiency

Cybersickness remains a critical barrier to the widespread adoption of Virtual Reality (VR), particularly in scenarios involving intense or artificial motion cues. Among the key contributors is excessive optical flow-perceived visual motion that, when unmatched by vestibular input, leads to sensory conflict and discomfort. While previous efforts have explored geometric or hardware based mitigation strategies, such methods often rely on predefined scene structures, manual tuning, or intrusive equipment. In this work, we propose U-MAD, a lightweight, real-time, AI-based solution that suppresses perceptually disruptive optical flow directly at the image level. Unlike prior handcrafted approaches, this method learns to attenuate high-intensity motion patterns from rendered frames without requiring mesh-level editing or scene specific adaptation. Designed as a plug and play module, U-MAD integrates seamlessly into existing VR pipelines and generalizes well to procedurally generated environments. The experiments show that U-MAD consistently reduces average optical flow and enhances temporal stability across diverse scenes. A user study further confirms that reducing visual motion leads to improved perceptual comfort and alleviated cybersickness symptoms. These findings demonstrate that perceptually guided modulation of optical flow provides an effective and scalable approach to creating more user-friendly immersive experiences. The code will be released at https://github.com/XXXXX (upon publication).

💡 Research Summary

The paper addresses the persistent problem of cybersickness in virtual reality (VR), which is largely driven by a mismatch between visual motion cues—especially high‑intensity optical flow—and vestibular input. Existing mitigation strategies, such as hardware‑based vestibular stimulation, geometric scene simplification, or peripheral visual anchoring, either require intrusive equipment, manual scene editing, or introduce undesirable visual artifacts. To overcome these limitations, the authors propose U‑MAD (U‑shaped Mamba Diffusion), a lightweight, real‑time, image‑level AI module that directly attenuates disruptive optical flow while preserving overall visual fidelity.

U‑MAD’s architecture consists of a U‑shaped encoder‑decoder built on the Mamba state‑space model, which efficiently captures long‑range temporal dependencies with lower computational cost than conventional Transformers. The input to the system is a sequence of degraded VR frames (F_deg) and, during training only, a down‑sampled clean reference (F_raw). The degraded frames are cropped into 512 × 512 patches for processing efficiency. Two auxiliary context pathways augment the local patch information: (1) a Global Context Module (GCM) that encodes a low‑resolution version of the full‑frame scene, providing global structural cues; and (2) a Post‑Temporal Context Module (PTCM) that attends to frames occurring after the target timestep, reinforcing short‑term motion consistency. Both modules employ lightweight Temporal‑Spatial Convolution (TSC) blocks, making the overall pipeline suitable for mobile or low‑power devices.

Crucially, optical flow between consecutive degraded frames is computed on‑the‑fly and passed through a dedicated flow encoder. The resulting motion embeddings are injected as conditional signals into the Mamba backbone, guiding the diffusion denoising process to produce temporally coherent reconstructions while selectively suppressing high‑magnitude flow. The loss function combines reconstruction (L2), perceptual (SSIM), and flow‑consistency (L1) terms, ensuring that the model reduces perceived motion without completely erasing necessary visual dynamics.

Quantitative experiments span a variety of VR environments, including procedurally generated scenes and high‑resolution (4K) content. Compared with baseline methods that either blur peripheral regions or perform geometric simplification, U‑MAD achieves an average reduction of 18 % in optical‑flow magnitude, while improving PSNR by 1.2 dB and SSIM by 0.03. Ablation studies demonstrate that removing either the GCM or the PTCM degrades both visual quality and flow‑suppression performance, confirming the importance of global and short‑term context.

A user study with 30 participants further validates the approach. Participants experienced a 22 % decrease in Simulator Sickness Questionnaire (SSQ) scores after U‑MAD processing, and reported higher comfort ratings (statistically significant, p < 0.01). Qualitative feedback highlighted that peripheral motion felt less “jarring” while overall immersion remained intact.

The authors acknowledge two primary limitations. First, training requires a clean high‑resolution reference, which may not be available in fully unsupervised deployment scenarios. Second, overly aggressive flow attenuation can lead to loss of peripheral detail, especially during rapid scene changes. Future work is suggested to integrate multimodal vestibular sensors for adaptive flow control, develop self‑supervised training pipelines, and explore user‑specific dynamic attenuation policies.

In summary, U‑MAD introduces a novel flow‑conditioned diffusion framework that operates in real time, is plug‑and‑play for existing VR pipelines, and demonstrably reduces cybersickness while maintaining visual quality. It represents a significant step toward scalable, perceptually‑aware VR rendering solutions.