TraceFlow: Dynamic 3D Reconstruction of Specular Scenes Driven by Ray Tracing

We present TraceFlow, a novel framework for high-fidelity rendering of dynamic specular scenes by addressing two key challenges: precise reflection direction estimation and physically accurate reflection modeling. To achieve this, we propose a Residual Material-Augmented 2D Gaussian Splatting representation that models dynamic geometry and material properties, allowing accurate reflection ray computation. Furthermore, we introduce a Dynamic Environment Gaussian and a hybrid rendering pipeline that decomposes rendering into diffuse and specular components, enabling physically grounded specular synthesis via rasterization and ray tracing. Finally, we devise a coarse-to-fine training strategy to improve optimization stability and promote physically meaningful decomposition. Extensive experiments on dynamic scene benchmarks demonstrate that TraceFlow outperforms prior methods both quantitatively and qualitatively, producing sharper and more realistic specular reflections in complex dynamic environments.

💡 Research Summary

TraceFlow introduces a novel framework for high‑fidelity reconstruction and rendering of dynamic specular scenes from monocular video. The core contribution lies in three tightly coupled components. First, the authors extend 2‑D Gaussian Splatting (2DGS) with a residual, time‑conditioned network that predicts per‑Gaussian offsets for position, scale, rotation, opacity, and a newly introduced specular‑tint parameter. This Residual Material‑Augmented 2DGS (RM‑2DGS) preserves the exact surface normals inherent to 2DGS—computed as the cross‑product of the two tangent vectors—thereby eliminating the approximations that plague 3‑D Gaussian or NeRF‑based methods. Accurate normals enable precise computation of the reflected ray direction via the classic law d_out = d_in − 2(d_in·n)n, which is essential for capturing high‑frequency specular details.

Second, TraceFlow models the surrounding illumination with a Dynamic Environment Gaussian (DEG) representation. Unlike static environment maps or spherical‑harmonic encodings, DEG learns a set of Gaussians that evolve over time, effectively acting as a learnable, temporally varying light probe. When a reflected ray, generated from the RM‑2DGS surface, intersects the DEG, the sampled color provides a physically grounded specular contribution. This design overcomes the resolution limits of traditional environment maps and captures near‑field reflections that are otherwise lost.

Third, the rendering pipeline is hybrid: diffuse appearance is obtained through the fast rasterization of 2DGS, while specular appearance is computed by tracing the reflected ray against the DEG using a lightweight ray‑tracer. The two components are blended to produce the final pixel color. To train this system robustly, the authors propose a coarse‑to‑fine schedule. Early epochs use low‑resolution images and sparse sampling to stabilize geometry learning; later stages progressively increase resolution and sample density, allowing the network to refine material and specular details.

Supervision includes a geometry‑aligned normal loss that enforces consistency between rendered normals and pseudo‑normals derived from depth gradients, as well as a temporal‑consistent normal loss leveraging NormalCrafter, a diffusion‑based normal estimator that supplies temporally stable normal priors. These losses mitigate the residual network’s tendency to drift and ensure temporally coherent surface orientation, which is critical for dynamic scenes.

Extensive experiments on benchmark dynamic specular datasets (e.g., “Plate” and other high‑gloss scenes) demonstrate that TraceFlow outperforms prior state‑of‑the‑art methods—including dynamic NeRF extensions, 7‑D Gaussian approaches, and previous dynamic environment‑map techniques—by notable margins (≈0.74 dB higher PSNR, +0.036 SSIM, and –0.031 LPIPS). Qualitatively, the method produces sharper highlights, clearer reflection ripples, and faithful color shifts as the environment changes. Performance-wise, the hybrid rasterization‑ray‑tracing pipeline achieves real‑time frame rates (>30 fps) on modern GPUs, with memory usage scaling linearly with the number of environment Gaussians.

The paper acknowledges limitations: handling multiple reflection bounces, transparent materials, and highly complex lighting would require additional ray tracing depth and could increase computational load. Moreover, the DEG representation, while more expressive than static maps, still incurs memory overhead as scene dynamics become more intricate. Future work is suggested to integrate multi‑bounce tracing, advanced transparency models, and GPU‑accelerated ray‑tracing cores to further push real‑time capabilities.

In summary, TraceFlow delivers a compelling solution that unifies accurate normal‑based reflection direction computation, temporally adaptive material modeling, and a physically grounded environment representation within an efficient hybrid rendering framework, setting a new benchmark for dynamic specular scene reconstruction.