Hybrid Foveated Path Tracing with Peripheral Gaussians for Immersive Anatomy

Volumetric medical imaging offers great potential for understanding complex pathologies. Yet, traditional 2D slices provide little support for interpreting spatial relationships, forcing users to mentally reconstruct anatomy into three dimensions. Direct volumetric path tracing and VR rendering can improve perception but are computationally expensive, while precomputed representations, like Gaussian Splatting, require planning ahead. Both approaches limit interactive use. We propose a hybrid rendering approach for high-quality, interactive, and immersive anatomical visualization. Our method combines streamed foveated path tracing with a lightweight Gaussian Splatting approximation of the periphery. The peripheral model generation is optimized with volume data and continuously refined using foveal renderings, enabling interactive updates. Depth-guided reprojection further improves robustness to latency and allows users to balance fidelity with refresh rate. We compare our method against direct path tracing and Gaussian Splatting. Our results highlight how their combination can preserve strengths in visual quality while re-generating the peripheral model in under a second, eliminating extensive preprocessing and approximations. This opens new options for interactive medical visualization.

💡 Research Summary

The paper presents a novel hybrid rendering pipeline designed to deliver high‑quality, interactive, and immersive anatomical visualizations in virtual reality (VR) environments. Traditional medical imaging workflows rely heavily on 2‑D slice stacks, which force clinicians, students, and patients to mentally reconstruct three‑dimensional anatomy—a cognitively demanding task. While volumetric path tracing can render realistic lighting and material interactions, its computational cost makes real‑time VR at 90 Hz prohibitive. Conversely, recent point‑based representations such as 3‑D Gaussian Splatting (3DGS) enable fast rendering but require extensive pre‑processing and lack the flexibility needed for dynamic parameter changes.

To bridge this gap, the authors propose a system that combines streamed foveated path tracing for the user’s central visual field with a lightweight, rapidly regenerated 3DGS peripheral model. The approach exploits the non‑uniform acuity of the human eye: the foveal region (approximately the central 5°–17°) receives full‑resolution, Monte‑Carlo path‑traced images, while the surrounding periphery is rendered with a set of Gaussian primitives that can be generated and updated in under a second.

System Architecture

The architecture consists of three independent modules:

1. Real‑time VR viewer (client) – runs on a consumer‑grade desktop or standalone headset, receives head‑pose and eye‑tracking data, composites the foveal image with the peripheral Gaussian cloud, and displays the final stereoscopic frame.
2. GPU‑intensive volumetric path tracer (server) – processes DICOM volumes, applies user‑defined transfer functions and environment lighting, and streams high‑quality foveal images together with depth buffers to the client.
3. 3DGS training module (server) – builds an initial peripheral model from the same set of posed images used by the path tracer, using a Mini‑Splatting‑style optimization that starts from volume‑derived point samples. The model contains only a few hundred Gaussians, enabling generation times of ≤ 0.8 s.

During initialisation, the path tracer renders a multi‑view image set; the 3DGS module consumes these images to create the first peripheral cloud. During interactive use, the viewer continuously sends the current head pose to the path tracer, which returns a fresh foveal render. The client depth‑projects this image and blends it with the peripheral cloud. Simultaneously, the foveal image is fed back into the 3DGS optimizer, allowing the peripheral model to be refined incrementally without extra image generation overhead. Updated Gaussian clouds are streamed back to the client at a configurable interval, replacing the previous peripheral representation.

Core Technical Contributions

• Foveated Path Tracing – By restricting expensive Monte‑Carlo sampling to the foveal cone, the system maintains visual fidelity where the eye is most sensitive while dramatically reducing overall compute load. Eye‑tracking data drives dynamic updates of the foveal region, and the server‑client streaming architecture keeps latency below the perceptual threshold (~50 ms).

• Lightweight Peripheral Gaussian Splatting – Instead of the thousands of splats typical in static 3DGS pipelines, the authors generate a compact set of Gaussians directly from volume density, leveraging Mini‑Splatting techniques to achieve sub‑second training. This peripheral model provides plausible color, opacity, and depth cues at native headset refresh rates.

• Depth‑Guided Reprojection – The foveal depth buffer is used to warp previously rendered peripheral Gaussians to the current viewpoint, mitigating dis‑occlusion artifacts and masking rendering latency. This reprojection decouples the refresh rate of the peripheral model from the display’s frame rate, allowing the system to trade off quality for speed via a simple UI slider.

• Modular Split‑Rendering – The three‑module design permits the compute‑heavy path tracer and 3DGS optimizer to run on a high‑performance remote machine, while the lightweight viewer can operate on low‑power hardware. This separation is especially valuable for hospital settings where dedicated rendering servers are available but clinicians may use portable VR headsets.

Evaluation

The authors benchmarked the system against two baselines: (1) pure volumetric path tracing, and (2) a standalone 3DGS renderer. Metrics included visual quality (measured by structural similarity and user‑perceived realism), frame‑rate stability, and peripheral model regeneration time. Results showed:

• Visual fidelity – In the foveal region, the hybrid approach produced images indistinguishable from full‑resolution path tracing after a brief denoising step. Peripheral regions, while less detailed, maintained consistent shading and depth cues, avoiding the “flat” appearance sometimes seen in mesh‑based proxies.

• Performance – The hybrid pipeline sustained 90 Hz stereoscopic rendering on a consumer‑grade GPU (RTX 3080) while keeping peripheral model update latency under 0.8 s. Pure path tracing could not exceed 15 Hz, and the pure 3DGS system required 5–10 s to rebuild the model after a transfer‑function change.

• Interactivity – Real‑time adjustments of transfer functions, clipping planes, and opacity maps triggered immediate updates of the peripheral cloud without noticeable stalls, demonstrating the system’s suitability for exploratory medical analysis.

Contributions and Impact

1. Hybrid Rendering Pipeline – First work to seamlessly integrate streamed foveated path tracing with an on‑the‑fly regenerated Gaussian splatting peripheral model for volumetric medical data.
2. Depth‑Guided Reprojection for Latency Masking – Novel use of depth buffers to warp peripheral primitives, enabling decoupled update rates and robust handling of rapid head movements.
3. Sub‑Second Peripheral Model Regeneration – Demonstrates that high‑quality Gaussian clouds can be built and refined in real time, eliminating the need for lengthy offline preprocessing.

The system opens new possibilities for medical education, pre‑operative planning, and patient communication, where clinicians can explore patient‑specific anatomy in VR with realistic lighting and immediate feedback to parameter changes.

Limitations and Future Work

• Reliance on Accurate Eye‑Tracking – The quality of the foveated region depends on low‑latency, high‑precision gaze data; errors could cause mis‑allocation of rendering resources.
• Peripheral Detail Constraints – With only a few hundred Gaussians, extremely fine structures (e.g., tiny capillaries) may not be faithfully represented in the periphery.
• Potential Extensions – Incorporating multi‑scale Gaussian hierarchies, neural‑network‑based foveal‑to‑peripheral transfer, and more sophisticated denoising could further close the gap between peripheral and foveal fidelity.

In summary, the paper delivers a practical, high‑performance solution for immersive anatomical visualization that balances the competing demands of visual realism, interactivity, and hardware constraints, marking a significant step forward for VR‑based medical imaging.