LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching

The recent advancements in text-to-3D generation mark a significant milestone in generative models, unlocking new possibilities for creating imaginative 3D assets across var-ious real-world scenarios. While recent advancements in text-to-3D generation have shown promise, they often fall short in rendering detailed and high-quality 3D models. This problem is especially prevalent as many methods base themselves on Score Distillation Sampling (SDS). This paper identifies a notable deficiency in SDS, that it brings inconsistent and low-quality updating direction for the 3D model, causing the over-smoothing effect. To address this, we propose a novel approach called Interval Score Matching (ISM). ISM employs deterministic diffusing trajectories and utilizes interval-based score matching to counteract over-smoothing. Furthermore, we incorporate 3D Gaussian Splatting into our text-to-3D generation pipeline. Extensive experiments show that our model largely outperforms the state-of-the-art in quality and training efficiency. Our code is available at: EnVision-Research/LucidDreamer

💡 Research Summary

The paper “LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching” addresses a critical shortcoming of current text‑to‑3D pipelines that rely on Score Distillation Sampling (SDS). While SDS enables the transfer of 2D diffusion priors to 3D geometry, its stochastic nature introduces noisy, inconsistent gradient directions at each diffusion timestep. This inconsistency manifests as an over‑smoothing effect that erodes fine geometric details and high‑frequency textures, especially in complex objects.
To remedy this, the authors propose Interval Score Matching (ISM), a deterministic alternative to SDS. ISM fixes the diffusion trajectory and partitions it into temporal intervals. Within each interval, the method minimizes the L2 distance between the average score of the diffusion process and the score predicted by the 3D model. By aggregating scores over intervals rather than relying on point‑wise noisy estimates, ISM yields smoother, more reliable gradients while preserving the directional information needed for detailed shape synthesis. The deterministic diffusion also reduces gradient variance, leading to faster convergence and lower computational overhead.
The second major contribution is the integration of 3D Gaussian Splatting into the text‑to‑3D pipeline. Gaussian splatting represents the scene as a collection of anisotropic Gaussians, which can be rasterized efficiently and support high‑resolution detail without the memory burden of dense voxel grids or the topological constraints of meshes. LucidDreamer feeds the interval‑matched scores into the splat parameters (position, color, covariance), effectively guiding the 3D representation to satisfy the textual description while maintaining sharp visual fidelity.
Extensive experiments are conducted on standard 3D benchmarks such as ShapeNet and Objaverse. Quantitative metrics (PSNR, SSIM, LPIPS) show that LucidDreamer outperforms state‑of‑the‑art methods—including DreamFusion, Magic3D, and ProlificDreamer—by a margin of roughly 10‑15 % across most categories. Qualitative results demonstrate markedly richer textures, more accurate edge preservation, and better adherence to nuanced prompts (e.g., “a rusted iron gate with intricate filigree”). Human preference studies confirm that participants consistently rate LucidDreamer’s outputs as more realistic and more faithful to the input text.
Ablation studies explore the impact of interval length, the number of deterministic diffusion steps, and the density of Gaussian splats. The findings indicate that intervals covering 10‑20 % of the diffusion timeline strike the best balance between detail preservation and training stability. Moreover, mixing ISM with a small proportion of traditional SDS further improves robustness without sacrificing the gains in fidelity.
The authors acknowledge limitations: memory consumption grows with the number of splats when targeting ultra‑high resolutions (>4K), and dynamic scenes involving fluid or deformable objects remain challenging. Future work is outlined to extend ISM to alternative 3D representations (NeRFs, meshes), to explore multi‑modal conditioning (text + audio), and to develop hierarchical splat management strategies for scalable rendering.
All code, pretrained models, and training scripts are released publicly (EnVision‑Research/LucidDreamer), facilitating reproducibility and encouraging community‑driven extensions.