Multi-Agent DRL for V2X Resource Allocation: Disentangling Challenges and Benchmarking Solutions

Multi-agent deep reinforcement learning (DRL) has emerged as a promising approach for radio resource allocation (RRA) in cellular vehicle-to-everything (C-V2X) networks. However, the multifaceted challenges inherent to multi-agent reinforcement learning (MARL) - including non-stationarity, coordination difficulty, large action spaces, partial observability, and limited robustness and generalization - are often intertwined, making it difficult to understand their individual impact on performance in vehicular environments. Moreover, existing studies typically rely on different baseline MARL algorithms, and a systematic comparison of their capabilities in addressing specific challenges in C-V2X RRA remains lacking. In this paper, we bridge this gap by formulating C-V2X RRA as a sequence of multi-agent interference games with progressively increasing complexity, each designed to isolate a key MARL challenge. Based on these formulations, we construct a suite of learning tasks that enable controlled evaluation of performance degradation attributable to each challenge. We further develop large-scale, diverse training and testing datasets using SUMO-generated highway traces to capture a wide range of vehicular topologies and corresponding interference patterns. Through extensive benchmarking of representative MARL algorithms, we identify policy robustness and generalization across diverse vehicular topologies as the dominant challenge in C-V2X RRA. We further show that, on the most challenging task, the best-performing actor-critic method outperforms the value-based approach by 42%. By emphasizing the need for zero-shot policy transfer to both seen and unseen topologies at runtime, and by open-sourcing the code, datasets, and interference-game benchmark suite, this work provides a systematic and reproducible foundation for evaluating and advancing MARL algorithms in vehicular networks.

💡 Research Summary

This paper addresses the challenging problem of radio resource allocation (RRA) in cellular vehicle‑to‑everything (C‑V2X) networks by formulating it as a series of multi‑agent interference games and systematically evaluating state‑of‑the‑art multi‑agent deep reinforcement learning (MARL) algorithms. The authors begin by highlighting that RRA is inherently a multi‑agent problem: multiple vehicle‑to‑vehicle (V2V) links must share uplink spectrum allocated to vehicle‑to‑infrastructure (V2I) links, and decisions must be made in a highly dynamic, partially observable environment. While deep reinforcement learning (DRL) has shown promise for such tasks, MARL introduces five intertwined challenges—non‑stationarity, coordination difficulty, large action spaces, partial observability, and limited robustness/generalization—that have not been individually quantified in vehicular contexts.

To isolate each challenge, the authors construct four interference‑game formulations of increasing complexity. Game 1 includes only non‑stationarity (single sub‑channel, fixed power). Game 2 adds a larger discrete action space (multiple sub‑channels and four power levels). Game 3 introduces partial observability by restricting each agent’s state to local measurements (its own position, queue length, and channel gains). Game 4 incorporates a wide variety of vehicle topologies generated by SUMO, thereby exposing the need for policies that generalize to unseen configurations. This progressive design enables controlled experiments where the performance drop caused by adding a specific challenge can be directly measured.

A large‑scale dataset is built using SUMO highway simulations that follow 3GPP TR 36.885 and ETSI TR 103 766 specifications for speed‑density relationships. Over 10 000 traces (8 000 for training, 2 000 for testing) are generated, covering diverse densities, speeds, and headway settings. For each trace, large‑scale fading, small‑scale fading, and interference matrices are pre‑computed and treated as a Markov process, providing realistic channel dynamics for the learning agents.

Eight representative MARL algorithms are benchmarked across the four games: value‑based independent learners (DQN, Double‑DQN), a value‑based CTDE method (QMIX), policy‑gradient (MAPPO), actor‑critic independent (PPO, IPPO), a multi‑agent actor‑critic with centralized training (MADDPG), and a hybrid DQN‑DDPG approach. Both independent learning (IL) and centralized training with decentralized execution (CTDE) paradigms are evaluated for each algorithm, using identical network architectures, learning rates, batch sizes, and episode lengths to ensure fairness.

Key findings are: (1) Robustness and generalization across vehicle topologies emerge as the dominant bottleneck; algorithms that perform well on seen topologies can degrade dramatically on unseen ones. (2) In the most complex game (Game 4), actor‑critic methods outperform value‑based methods by an average of 42 % in terms of successful CAM delivery and overall throughput. (3) While CTDE provides modest gains in coordination, the independent PPO (IPPO) variant achieves comparable or better performance with far lower computational overhead and superior scalability to hundreds of agents. (4) Zero‑shot policy transfer—deploying a policy trained on a set of topologies directly to new, unseen topologies—requires training data with high topological diversity; the authors suggest meta‑learning or domain‑adaptation techniques as promising future directions.

The paper also contributes an open‑source repository containing the code, the SUMO‑generated datasets, and the full benchmark suite, enabling reproducible research and providing a common platform for future MARL developments in vehicular networks. In conclusion, the work offers a rigorous methodology for disentangling MARL challenges, demonstrates that actor‑critic approaches (especially IPPO) are currently the most effective for C‑V2X RRA, and underscores the critical need for policies that can generalize zero‑shot across diverse vehicular topologies.