TR-Gaussians: High-fidelity Real-time Rendering of Planar Transmission and Reflection with 3D Gaussian Splatting
📝 Abstract
We propose Transmission-Reflection Gaussians (TR-Gaussians), a novel 3D-Gaussian-based representation for high-fidelity rendering of planar transmission and reflection, which are ubiquitous in indoor scenes. Our method combines 3D Gaussians with learnable reflection planes that explicitly model the glass planes with view-dependent reflectance strengths. Real scenes and transmission components are modeled by 3D Gaussians and the reflection components are modeled by the mirrored Gaussians with respect to the reflection plane. The transmission and reflection components are blended according to a Fresnel-based, view-dependent weighting scheme, allowing for faithful synthesis of complex appearance effects under varying viewpoints. To effectively optimize TR-Gaussians, we develop a multi-stage optimization framework incorporating color and geometry constraints and an opacity perturbation mechanism. Experiments on different datasets demonstrate that TR-Gaussians achieve real-time, high-fidelity novel view synthesis in scenes with planar transmission and reflection, and outperform state-of-the-art approaches both quantitatively and qualitatively.
💡 Analysis
We propose Transmission-Reflection Gaussians (TR-Gaussians), a novel 3D-Gaussian-based representation for high-fidelity rendering of planar transmission and reflection, which are ubiquitous in indoor scenes. Our method combines 3D Gaussians with learnable reflection planes that explicitly model the glass planes with view-dependent reflectance strengths. Real scenes and transmission components are modeled by 3D Gaussians and the reflection components are modeled by the mirrored Gaussians with respect to the reflection plane. The transmission and reflection components are blended according to a Fresnel-based, view-dependent weighting scheme, allowing for faithful synthesis of complex appearance effects under varying viewpoints. To effectively optimize TR-Gaussians, we develop a multi-stage optimization framework incorporating color and geometry constraints and an opacity perturbation mechanism. Experiments on different datasets demonstrate that TR-Gaussians achieve real-time, high-fidelity novel view synthesis in scenes with planar transmission and reflection, and outperform state-of-the-art approaches both quantitatively and qualitatively.
📄 Content
T RANSPARENT glass panes (e.g., windows and show- cases) are ubiquitous in indoor scenes in people’s daily life, exhibiting complex combinations of transmission and reflection at their surfaces. It is of critical importance to accurately model such optical phenomena in photorealistic novel view synthesis (NVS) of indoor scenes. While there has been a rapid development of NVS in recent years, where 3D Gaussian Splatting (3DGS) [1] has emerged as the state-ofthe-art method, demonstrating high rendering quality in many types of scenes, it still struggles to faithfully reconstruct indoor scenes with glass panes. The fundamental reason is, given the training images with both transmission and reflection effects, 3DGS simply overfits them using low-opacity Gaussians, without correctly distinguishing or modeling the two components separately. As a result, while training views may be fitted well, under test views the reflection often exhibits noisy artifacts (e.g., Bookcase2 in Fig. 3) and can be completely missing in regions unobserved by training views (e.g., Loft in Fig. 3).
To the best of our knowledge, there has not been 3DGSrelated works studying the high-fidelity novel-view rendering of transparent glass panes in indoor scenes. The most relevant work is MirrorGaussian [2], which focuses only on mirrors with pure reflection without considering transmission. Recently several NeRF-based methods were proposed aiming to address the mixed phenomenon of transmission and reflection. They decompose the refection and transmission either by regarding the scene as a single shell without multiple surfaces [3], or by suppressing gradient similarities between primary and reflected colors [4], which only holds on limited views or lacking robustness on textureless regions. Moreover, these methods inevitably suffer from NeRF’s inherent computational inefficiency.
In this paper, we propose a novel representation of 3D Gaussians named Transmission-Reflection Gaussians (TR-Gaussians) enabling high-fidelity modeling of light transmission and reflection on transparent glass panes, which can be rendered in real time during novel view synthesis. TR-Gaussians consist of a set of primary Gaussians representing the real scene, a reflection plane representing the glass pane and modeling the view-dependent reflection strengths, and a set of glass Gaussians marking the reflective regions on the glass pane. We apply the Fresnel reflectance model on the reflection plane, where the reflection plane is associated with the learnable properties of position, orientation and base reflectance, and the reflection strength on each point of the plane is computed as the ratio of reflected light intensity relative to the incident light intensity based on these properties. The rendering of TR-Gaussians involves two passes. In the first pass, the primary Gaussians are used to render the real scene and the transmission image, and the primary and glass Gaussians are utilized together to render a reflection mask on the reflection plane marking the arbitrarily shaped reflective regions. We also use the reflection plane to compute the reflection strength map and filter it with the reflection mask. In the second pass, by leveraging the symmetry of planar reflections, we generate the mirrored Gaussians by reflecting the primary Gaussians about the reflection plane, and the mirrored Gaussians are rasterized to obtain the reflection image. The final image is obtained by blending the transmission image and reflection image based on the reflection strength map.
We design effective optimization strategies to optimize TR-Gaussians. First, to restrain the primary Gaussians from wrongly fitting the reflective regions with low-opacity Gaussians, which are commonly floating far from the real surfaces (see Fig. 2 for example), we introduce a depth variance loss to enforce the primary Gaussians to be distributed close to the surface, by minimizing the distances between the primary Gaussians’ center depths and the primary Gaussian rendered depths. By combining the depth variance loss with the image loss [1] and the gradient conflict loss [4], it is supposed that we can correctly decompose the reflection regions and real scenes. However, we find that relying solely on loss functions for decomposition may still get stuck in local optima. To this end, we propose a multi-stage optimization framework. In the first stage, all Gaussians are optimized together as vanilla 3DGS and then the reflection planes and glass Gaussians are initialized. In the second stage we start the joint optimization of the reflection planes along with the primary and glass Gaussians, but ignore the gradients from the reflection images to primary Gaussians so as to avoid the interference from reflection colors, because we find incorporating reflection colors together will yield wrongly placed reflection planes, which will cause the reflection cannot be correctly modeled and is degenerated to vanilla 3DG
This content is AI-processed based on ArXiv data.