An Agent-based Modelling Framework for Driving Policy Learning in Connected and Autonomous Vehicles

Abstract—Due to the complexity of the natural world, a programmer cannot foresee all possible situations, a connected and autonomous vehicle (CAV) will face during its operation, and hence, CAVs will need to learn to make decisions autonomously. Due to the sensing of its surroundings and information exchanged with other vehicles and road infrastructure, a CAV will have access to large amounts of useful data. While different control algorithms have been proposed for CAVs, the benefits brought about by connectedness of autonomous vehicles to other vehicles and to the infrastructure, and its implications on policy learning has not been investigated in literature. This paper investigates a data driven driving policy learning framework through an agent-based modelling approaches. The contributions of the paper are two-fold. A dynamic programming framework is proposed for in-vehicle policy learning with and without connectivity to neighboring vehicles. The simulation results indicate that while a CAV can learn to make autonomous decisions, vehicle-to-vehicle (V2V) communication of information improves this capability. Furthermore, to overcome the limitations of sensing in a CAV, the paper proposes a novel concept for infrastructure-led policy learning and communication with autonomous vehicles. In infrastructure-led policy learning, road-side infrastructure senses and captures successful vehicle maneuvers and learns an optimal policy from those temporal sequences, and when a vehicle approaches the road-side unit, the policy is communicated to the CAV. Deep-imitation learning methodology is proposed to develop such an infrastructure-led policy learning framework. Keywords—Agent-based learning; reinforcement learning; driving policy; data driven control; imitation learning I. INTRODUCTION The emergence of connected and autonomous vehicles (CAVs) marks a new phase of innovation in automotive and transportation industries since the development of personal automobiles [1]. Once technologically matured, CAVs would be able to move autonomously without the aid of a human driver and be able to communicate with other vehicles and traffic infrastructure. The promised benefits of CAVs are numerous, such as, reduced congestion, better fuel efficiency, reduced environmental pollution, reduced number of traffic related casualties and increased personal independence. The continuing evolution of sensor technologies, high speed communication infrastructure, machine learning and artificial intelligence will eventually bring us the above benefits. A CAV is composed of four major technological components. The first is the perception system, which is responsible for sensing the environment to understand its surroundings. The second component is the localization and mapping system that enables the vehicle to know its current location. The third component is responsible for the driving policy. The driving policy refers to the decision making capability of a CAV under various situations, such as negotiating at roundabouts, giving way to vehicles and pedestrians, and overtaking vehicles. Finally, the CAVs will be connected. It is expected that the CAVs will be connected to the surrounding vehicles: vehicle to vehicle connectivity (V2V), to the infrastructure: Vehicle to Infrastructure (V2I) and to anything else such as the internet: Vehicle to Anything (V2X), through wireless communications links [2]. While there are many challenges still to be addressed for high speed wireless connectivity for vehicular applications [3], the IEEE802.11p Wireless Access in Vehicular Environments (Wave) is considered the most relevant standard that currently caters to the requirements of such applications [4]. The connectedness of the CAV’s can be useful for many important functions related to intelligent mobility. One important use case is to exchange sensor data between vehicles for improved perception of the surroundings [5], which enables to reduce the accidents. Connected vehicles also enables centralized traffic control to ease congestion in smart city applications [6]. For example in Compass4D, a pilot project funded by the EU, it was demonstrated that 15% reductions in fuel efficiency can be achieved through V2I communication to control the flow of traffic [7]. Another interesting application of connectivity is in “vehicle platooning”, where a group of vehicles with common interests maintain a small and constant distance to each other [8]. Connectivity is also a key enable

…(Full text truncated)…