Bridging the Modality Gap in Roadside LiDAR: A Training-Free Vision-Language Model Framework for Vehicle Classification

Fine-grained truck classification is critical for intelligent transportation systems (ITS), yet current LiDAR-based methods face scalability challenges due to their reliance on supervised deep learning and labor-intensive manual annotation. Vision-Language Models (VLMs) offer promising few-shot generalization, but their application to roadside LiDAR is limited by a modality gap between sparse 3D point clouds and dense 2D imagery. We propose a framework that bridges this gap by adapting off-the-shelf VLMs for fine-grained truck classification without parameter fine-tuning. Our new depth-aware image generation pipeline applies noise removal, spatial and temporal registration, orientation rectification, morphological operations, and anisotropic smoothing to transform sparse, occluded LiDAR scans into depth-encoded 2D visual proxies. Validated on a real-world dataset of 20 vehicle classes, our approach achieves competitive classification accuracy with as few as 16-30 examples per class, offering a scalable alternative to data-intensive supervised baselines. We further observe a “Semantic Anchor” effect: text-based guidance regularizes performance in ultra-low-shot regimes $k < 4$, but degrades accuracy in more-shot settings due to semantic mismatch. Furthermore, we demonstrate the efficacy of this framework as a Cold Start strategy, using VLM-generated labels to bootstrap lightweight supervised models. Notably, the few-shot VLM-based model achieves over correct classification rate of 75 percent for specific drayage categories (20ft, 40ft, and 53ft containers) entirely without the costly training or fine-tuning, significantly reducing the intensive demands of initial manual labeling, thus achieving a method of practical use in ITS applications.

💡 Research Summary

The paper tackles the long‑standing scalability problem of roadside LiDAR‑based truck classification by leveraging pre‑trained vision‑language models (VLMs) without any parameter fine‑tuning. Traditional LiDAR approaches require large, manually annotated point‑cloud datasets for each deployment site, which is costly and time‑consuming. The authors identify two fundamental gaps that prevent direct use of VLMs: (1) a Modality Gap—VLM visual encoders are trained on dense, texture‑rich RGB images, whereas raw LiDAR point clouds are sparse, textureless, and often capture only a “half‑shell” of a vehicle; (2) a Reality‑to‑Sim Gap—most 3D‑VLMs are trained on complete CAD models, not on the partial, occluded scans obtained from roadside sensors.

To bridge these gaps, the authors propose a two‑stage pipeline. First, a Depth‑Aware Image Generation module converts raw LiDAR sequences into high‑fidelity 2D images that encode depth as color. The conversion pipeline includes voxel down‑sampling (0.05 m voxels), statistical outlier removal (k‑nearest neighbours, α = 1.0), temporal registration and tracking (SORT), orientation rectification, morphological operations (erosion/dilation), and anisotropic smoothing. These steps mitigate sensor sparsity, remove noise, and produce dense depth maps that resemble natural images in distribution, making them suitable for VLM encoders.

Second, the generated images are fed to off‑the‑shelf VLMs (CLIP with ViT‑B/32 and ViT‑L/14, and EVA‑L) together with domain‑specific textual prompts such as “a photo of a 20ft container truck”. Using in‑context few‑shot learning, the model computes cosine similarity between image embeddings and text embeddings to predict the class. No gradient updates are performed; the system operates in a purely inference‑only mode.

The authors evaluate the framework on a real‑world dataset comprising 20 fine‑grained truck classes (including bobtails, multi‑trailer configurations, and 20 ft/40 ft/53 ft containers). With only 16–30 labeled examples per class, the VLM‑based method achieves competitive accuracy, surpassing 75 % correct classification for the container categories. In comparison, a supervised ViT‑B/16 model trained on the same limited data reaches only ~55 % accuracy, and larger supervised models improve modestly but still lag behind the VLM approach. Inference speed is slower than lightweight supervised models, but the elimination of any training phase dramatically reduces deployment time and labeling cost.

A notable finding is the “Semantic Anchor” effect. In ultra‑low‑shot regimes (k < 4), adding textual prompts improves performance by 5–7 % because the text provides regularization when visual information is scarce. However, as the number of shots increases, the same textual guidance begins to degrade performance due to semantic mismatch between generic web‑scale image‑text pre‑training and the specialized engineering terminology of truck classes. This suggests that text‑based regularization should be applied selectively based on data availability.

The paper also demonstrates a Cold‑Start strategy: VLM‑generated pseudo‑labels are used to bootstrap lightweight supervised models (e.g., PointNet, MobileNet‑V2). This approach reduces the manual labeling burden by over 90 % while still delivering real‑time inference capability, making it attractive for rapid field deployment.

Key contributions are:

1. A novel depth‑aware image generation pipeline that effectively bridges the modality gap for roadside LiDAR.
2. Demonstration that training‑free few‑shot VLM inference can match or exceed supervised baselines with only a few dozen examples per class.
3. Systematic analysis of the trade‑off between visual and textual cues, revealing the Semantic Anchor phenomenon.
4. Empirical evidence that VLM‑generated labels can serve as a practical cold‑start for lightweight supervised models, enabling scalable ITS deployments.

Limitations include reliance on accurate background subtraction and vehicle segmentation, potential performance drops in highly congested scenes, and the need for prompt engineering to mitigate the semantic mismatch. Future work may explore adaptive prompt generation, real‑time depth‑aware rendering, and integration with multi‑sensor fusion to further improve robustness.

In summary, the study provides a practical, data‑efficient solution for fine‑grained truck classification using roadside LiDAR, showing that pre‑trained vision‑language models, when supplied with carefully crafted depth‑encoded visual proxies, can operate effectively without any additional training, thereby offering a scalable path forward for intelligent transportation systems.