A Self-Conditioned Representation Guided Diffusion Model for Realistic Text-to-LiDAR Scene Generation

Text-to-LiDAR generation can customize 3D data with rich structures and diverse scenes for downstream tasks. However, the scarcity of Text-LiDAR pairs often causes insufficient training priors, generating overly smooth 3D scenes. Moreover, low-quality text descriptions may degrade generation quality and controllability. In this paper, we propose a Text-to-LiDAR Diffusion Model for scene generation, named T2LDM, with a Self-Conditioned Representation Guidance (SCRG). Specifically, SCRG, by aligning to the real representations, provides the soft supervision with reconstruction details for the Denoising Network (DN) in training, while decoupled in inference. In this way, T2LDM can perceive rich geometric structures from data distribution, generating detailed objects in scenes. Meanwhile, we construct a content-composable Text-LiDAR benchmark, T2nuScenes, along with a controllability metric. Based on this, we analyze the effects of different text prompts for LiDAR generation quality and controllability, providing practical prompt paradigms and insights. Furthermore, a directional position prior is designed to mitigate street distortion, further improving scene fidelity. Additionally, by learning a conditional encoder via frozen DN, T2LDM can support multiple conditional tasks, including Sparse-to-Dense, Dense-to-Sparse, and Semantic-to-LiDAR generation. Extensive experiments in unconditional and conditional generation demonstrate that T2LDM outperforms existing methods, achieving state-of-the-art scene generation.

💡 Research Summary

This paper introduces T2LDM, a novel Text-to-LiDAR Diffusion Model designed to generate realistic and controllable 3D LiDAR scenes from natural language descriptions. The primary challenge addressed is the scarcity of high-quality Text-LiDAR paired data, which often leads to overly smooth and detail-lacking generation results in prior work.

The core innovation is a Self-Conditioned Representation Guidance (SCRG) mechanism. SCRG employs an auxiliary Guidance Network (GN) that operates during training to provide multi-scale, geometrically detailed supervisory signals to the main Denoising Network (DN). The GN aligns its outputs with “real representations,” effectively teaching the DN to recover finer structural details from the data distribution. Crucially, the GN is decoupled during inference, eliminating any additional computational cost while preserving the gained fidelity in generated scenes with detailed objects.

To enable rigorous evaluation and study of text-guided generation, the authors construct T2nuScenes, a content-composable Text-LiDAR benchmark based on the nuScenes dataset. It features re-annotated, flexible textual descriptions at both object-level (e.g., number, location of cars) and scene-level (e.g., weather, time). A key contribution is the introduction of a Text-Box Matching Rate (TBR) metric, which uses an off-the-shelf 3D detector to quantitatively evaluate how well the generated scene content aligns with the input text prompt. Using this benchmark, the paper presents a comprehensive analysis of how text prompt forms—length, specificity, complexity—affect generation quality and controllability, offering practical insights and prompt templates (e.g., “weather, location”) for optimal results.

Another significant contribution is the Directional Position Encoding (DPE), which tackles a structural distortion problem inherent in LiDAR generation. The common practice of projecting 3D LiDAR points onto a 2D range map via spherical projection can cause directional confusion (e.g., a “front-right” object appearing on the “left” of the map). This leads to unrealistic artifacts like bent or broken roads in generated scenes. DPE explicitly encodes the true horizontal and vertical angles for each pixel in the range map, providing the model with accurate directional priors and significantly improving spatial fidelity.

Furthermore, the framework demonstrates impressive versatility by extending to multiple conditional generation tasks via a ControlNet-like conditional encoder attached to the frozen pre-trained DN. This allows T2LDM to perform Sparse-to-Dense, Dense-to-Sparse, and Semantic-to-LiDAR generation effectively.

Extensive experiments on both unconditional and conditional LiDAR scene generation demonstrate that T2LDM outperforms all existing methods, achieving state-of-the-art results in terms of visual realism, structural detail, and adherence to textual or other input conditions.