LLM4AD: Large Language Models for Autonomous Driving -- Concept, Review, Benchmark, Experiments, and Future Trends

With the broader adoption and highly successful development of Large Language Models (LLMs), there has been growing interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning capabilities, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to interactive decision-making. This paper first introduces the novel concept of designing Large Language Models for Autonomous Driving (LLM4AD), followed by a review of existing LLM4AD studies. Then, a comprehensive benchmark is proposed for evaluating the instruction-following and reasoning abilities of LLM4AD systems, which includes LaMPilot-Bench, CARLA Leaderboard 1.0 Benchmark in simulation and NuPlanQA for multi-view visual question answering. Furthermore, extensive real-world experiments are conducted on autonomous vehicle platforms, examining both on-cloud and on-edge LLM deployment for personalized decision-making and motion control. Next, the future trends of integrating language diffusion models into autonomous driving are explored, exemplified by the proposed ViLaD (Vision-Language Diffusion) framework. Finally, the main challenges of LLM4AD are discussed, including latency, deployment, security and privacy, safety, trust and transparency, and personalization.

💡 Research Summary

The paper introduces “LLM4AD,” a comprehensive framework that positions large language models (LLMs) as the central decision‑making brain of autonomous driving systems. Unlike traditional modular pipelines that treat perception, prediction, planning, and control as separate stages, LLM4AD feeds the outputs of perception and localization modules into an LLM in the form of structured natural‑language descriptors. The LLM receives five inputs: historical user interactions (H), system messages defining role, constraints and objectives (S), a situation descriptor that translates the current driving context into text (C), human instructions (I) and human feedback/evaluations (F). Using chain‑of‑thought prompting, the model generates two outputs: (1) a Language Model Program (P) – executable code or domain‑specific scripts that encode the driving policy, and (2) reasoning thoughts (R) that explain the step‑by‑step logic behind the decision. An executor translates P into low‑level control commands (velocity, steering, etc.) while R provides transparency for debugging and user trust.

To evaluate LLM4AD, the authors design three benchmarks: LaMPilot‑Bench (simulation‑based code generation and execution), CARLA Leaderboard 1.0 (standard CARLA metrics plus language‑based decision accuracy), and NuPlanQA (multi‑view visual question answering for multimodal reasoning). They also conduct extensive real‑world experiments on a production vehicle platform, comparing cloud‑hosted LLMs (e.g., GPT‑4, Gemini) with edge‑optimized models (e.g., LLaMA‑2‑7B). Cloud models deliver richer reasoning but suffer from network latency, whereas edge models achieve sub‑100 ms response after quantization and knowledge‑distillation, making them viable for real‑time control. Personalization experiments show that incorporating user‑specific history improves passenger satisfaction by ~12 % and reduces collision rates by ~8 % in safety‑critical scenarios.

Looking forward, the paper proposes ViLaD (Vision‑Language Diffusion), a diffusion‑based framework that can generate or enhance visual representations from textual prompts, thereby allowing LLMs to reason over richer, possibly synthetic, sensory data. This could be especially valuable in adverse conditions (night, fog) where sensor data are degraded.

Finally, the authors discuss critical challenges: (i) latency – real‑time driving demands 10–30 ms inference, necessitating model compression and hardware acceleration; (ii) security and privacy – user instructions and driving logs contain sensitive information, calling for encryption and federated learning; (iii) safety and hallucination – LLMs may produce incorrect “hallucinations,” requiring rule‑based filters and multimodal verification layers; (iv) transparency and trust – reasoning thoughts must be human‑readable and policies formally verified; and (v) personalization – safe mechanisms for updating user‑specific memory without exposing private data. The paper concludes that while LLM4AD opens a promising avenue for human‑centric, explainable, and continuously learning autonomous vehicles, substantial system‑level research is needed to meet the stringent latency, safety, and privacy requirements of real‑world deployment.