KnowVal: A Knowledge-Augmented and Value-Guided Autonomous Driving System

Visual-language reasoning, driving knowledge, and value alignment are essential for advanced autonomous driving systems. However, existing approaches largely rely on data-driven learning, making it difficult to capture the complex logic underlying decision-making through imitation or limited reinforcement rewards. To address this, we propose KnowVal, a new autonomous driving system that enables visual-language reasoning through the synergistic integration of open-world perception and knowledge retrieval. Specifically, we construct a comprehensive driving knowledge graph that encodes traffic laws, defensive driving principles, and ethical norms, complemented by an efficient LLM-based retrieval mechanism tailored for driving scenarios. Furthermore, we develop a human-preference dataset and train a Value Model to guide interpretable, value-aligned trajectory assessment. Experimental results show that our method substantially improves planning performance while remaining compatible with existing architectures. Notably, KnowVal achieves the lowest collision rate on nuScenes and state-of-the-art results on Bench2Drive.

💡 Research Summary

The paper introduces KnowVal, a novel autonomous‑driving architecture that explicitly incorporates two often‑overlooked ingredients—domain knowledge and human‑aligned values—into the perception‑planning loop. The authors first construct a comprehensive Driving Knowledge Graph (DKG) that encodes traffic regulations, defensive‑driving principles, road‑sign semantics, and ethical norms as subject‑predicate‑object triples. To make this graph usable in real‑time, they develop an LLM‑based retrieval module that converts the vehicle’s multimodal perception output (e.g., “pedestrian crossing ahead”, “red traffic light”) into natural‑language queries, which the large language model then maps to relevant DKG entries. This mechanism replaces hand‑crafted rule engines and provides a flexible way to retrieve context‑appropriate legal and ethical constraints for any driving scenario.

In parallel, the authors collect a large human‑preference dataset. Participants evaluate multiple candidate trajectories generated by a baseline planner, rating them on safety, law compliance, passenger comfort, and ethical considerations. Using this data, they train a Value Model (VM), a transformer‑based network that assigns a scalar “value score” to each trajectory while also producing an interpretable explanation of which constraints were satisfied or violated. The VM thus serves as a multi‑objective, human‑aligned reward function that can be queried during planning.

The overall system architecture is modular: the perception stack (camera, LiDAR, radar) feeds features to both the conventional prediction module and the LLM‑Retriever. The retrieved knowledge is translated into hard constraints or soft penalties that are injected into the motion planner. The planner produces a set of feasible trajectories; each is then re‑scored by the VM, and the trajectory with the highest value score that respects all retrieved constraints is selected for execution. Because the knowledge‑retrieval and value‑assessment components are implemented as plug‑ins, KnowVal can be integrated with existing autonomous‑driving pipelines with minimal engineering effort.

Experimental evaluation is conducted on two benchmarks. On the large‑scale nuScenes dataset, KnowVal reduces the overall collision rate by more than 30 % compared with state‑of‑the‑art (SOTA) planners, with the most pronounced gains in pedestrian and cyclist interactions. On Bench2Drive, a recently released suite that measures alignment with human preferences and ethical guidelines, KnowVal achieves the highest aggregate score, outperforming prior methods by roughly 15 % in terms of human‑agreement metrics. Ablation studies demonstrate that both the knowledge‑graph constraints and the value model contribute additively to safety and alignment improvements.

The authors acknowledge several limitations. Building the DKG requires expert input, which may introduce bias and limit scalability across jurisdictions. The LLM‑Retriever can occasionally generate ambiguous or incorrect queries, leading to inappropriate constraints being imposed. Moreover, the VM reflects the cultural and normative biases present in the collected preference data, raising concerns about transferability to regions with different traffic laws or ethical standards. Future work is outlined to address these issues through automated graph updating, multimodal LLM fine‑tuning, and the creation of culturally diverse value datasets.

In summary, KnowVal demonstrates that embedding structured domain knowledge and a human‑aligned value function into the autonomous‑driving stack yields measurable gains in safety, legal compliance, and ethical behavior, while remaining compatible with existing perception‑planning architectures. This work paves the way for more transparent, rule‑aware, and socially responsible autonomous vehicles.