All You Need for Object Detection: From Pixels, Points, and Prompts to Next-Gen Fusion and Multimodal LLMs/VLMs in Autonomous Vehicles

Autonomous Vehicles (AVs) are transforming the future of transportation through advances in intelligent perception, decision-making, and control systems. However, their success is tied to one core capability, reliable object detection in complex and multimodal environments. While recent breakthroughs in Computer Vision (CV) and Artificial Intelligence (AI) have driven remarkable progress, the field still faces a critical challenge as knowledge remains fragmented across multimodal perception, contextual reasoning, and cooperative intelligence. This survey bridges that gap by delivering a forward-looking analysis of object detection in AVs, emphasizing emerging paradigms such as Vision-Language Models (VLMs), Large Language Models (LLMs), and Generative AI rather than re-examining outdated techniques. We begin by systematically reviewing the fundamental spectrum of AV sensors (camera, ultrasonic, LiDAR, and Radar) and their fusion strategies, highlighting not only their capabilities and limitations in dynamic driving environments but also their potential to integrate with recent advances in LLM/VLM-driven perception frameworks. Next, we introduce a structured categorization of AV datasets that moves beyond simple collections, positioning ego-vehicle, infrastructure-based, and cooperative datasets (e.g., V2V, V2I, V2X, I2I), followed by a cross-analysis of data structures and characteristics. Ultimately, we analyze cutting-edge detection methodologies, ranging from 2D and 3D pipelines to hybrid sensor fusion, with particular attention to emerging transformer-driven approaches powered by Vision Transformers (ViTs), Large and Small Language Models (SLMs), and VLMs. By synthesizing these perspectives, our survey delivers a clear roadmap of current capabilities, open challenges, and future opportunities.

💡 Research Summary

This survey provides a forward‑looking, comprehensive review of object detection for autonomous vehicles (AVs) with a strong emphasis on multimodal perception, sensor fusion, and the emerging integration of large language models (LLMs) and vision‑language models (VLMs). The authors begin by cataloguing the four primary sensor families—cameras, LiDAR, radar, and ultrasonic—detailing their physical principles, strengths, and failure modes. Cameras deliver high‑resolution color and texture cues but suffer under adverse illumination and weather; LiDAR offers precise 3‑D geometry yet is limited by reflectivity and point density; radar excels in long‑range and poor‑visibility conditions but provides coarse spatial resolution; ultrasonic sensors are valuable for very close‑range obstacle detection but have a narrow field of view. Recognizing the complementary nature of these modalities, the paper outlines a three‑stage fusion taxonomy: raw‑data level synchronization and calibration, feature‑level cross‑modal attention and alignment, and decision‑level probabilistic or deep‑ensemble integration. Transformer‑based Fusion Transformers and multi‑scale attention mechanisms are highlighted as the most effective tools for aligning spatiotemporal information across modalities.

The survey then introduces a novel taxonomy for AV datasets that goes beyond traditional ego‑vehicle collections (e.g., KITTI, Waymo Open, nuScenes). It classifies datasets into ego‑vehicle, infrastructure‑based, and cooperative (V2V, V2I, I2I) categories, describing their annotation formats (2‑D boxes, 3‑D cuboids, semantic maps, language captions) and statistical properties such as long‑tail distributions and rarity of edge‑case events. The authors discuss synthetic data generation and augmentation strategies, noting the persistent domain‑shift challenges when transferring learned representations to real‑world scenarios.

In the methodology section, the paper surveys the evolution from classic 2‑D detectors (Faster R‑CNN, SSD) and 3‑D LiDAR pipelines (PointRCNN, PV‑RCNN) to hybrid 2‑D/3‑D fusion architectures. Recent transformer‑based detectors such as ViT‑Det, DETR, and Deformable DETR are examined, with particular attention to how they incorporate multimodal attention to fuse visual and depth cues efficiently. The authors also compare large‑scale and lightweight models, analyzing the trade‑offs between parameter count, inference latency, and detection accuracy.

A central contribution of the survey is the systematic analysis of LLM and VLM integration. Two primary roles are identified: (1) language‑driven prompting and explanation, where LLMs translate sensor data and detection outcomes into natural‑language descriptions, enabling transparent human‑vehicle interaction and facilitating debugging; (2) zero‑shot or few‑shot detection via pretrained vision‑language embeddings, allowing the system to recognize novel objects or rare scenarios (e.g., unusual road constructions) without explicit retraining. The paper evaluates the feasibility of coupling small language models (SLMs) for on‑device inference with larger LLMs (e.g., GPT‑4) for cloud‑based reasoning, and discusses lightweight VLM variants (e.g., CLIP‑Tiny) that meet real‑time constraints.

Finally, the authors outline open research challenges: (i) real‑time optimization of multimodal attention mechanisms and hardware acceleration; (ii) seamless integration of LLM/VLM reasoning into the perception‑planning‑control loop; (iii) security, privacy, and robustness in V2X cooperative perception; (iv) scalable, automated data pipelines that continuously incorporate new sensor modalities and language annotations; and (v) the development of standardized APIs and open‑source frameworks to foster reproducibility. By synthesizing sensor technology, dataset taxonomy, detection algorithms, and generative AI advances, the survey provides a clear roadmap for researchers and practitioners aiming to build safer, more reliable, and explainable autonomous driving systems.