Asynchronous MultiAgent Reinforcement Learning for 5G Routing under Side Constraints

Networks in the current 5G and beyond systems increasingly carry heterogeneous traffic with diverse quality-of-service constraints, making real-time routing decisions both complex and time-critical. A common approach, such as a heuristic with human intervention or training a single centralized RL policy or synchronizing updates across multiple learners, struggles with scalability and straggler effects. We address this by proposing an asynchronous multi-agent reinforcement learning (AMARL) framework in which independent PPO agents, one per service, plan routes in parallel and commit resource deltas to a shared global resource environment. This coordination by state preserves feasibility across services and enables specialization for service-specific objectives. We evaluate the method on an O-RAN like network simulation using nearly real-time traffic data from the city of Montreal. We compared against a single-agent PPO baseline. AMARL achieves a similar Grade of Service (acceptance rate) (GoS) and end-to-end latency, with reduced training wall-clock time and improved robustness to demand shifts. These results suggest that asynchronous, service-specialized agents provide a scalable and practical approach to distributed routing, with applicability extending beyond the O-RAN domain.

💡 Research Summary

The paper tackles the pressing challenge of real‑time routing in 5G and beyond networks, where heterogeneous services such as eMBB, URLLC, and massive IoT must share limited link and compute resources while meeting strict latency and bandwidth guarantees. Traditional solutions—human‑tuned heuristics, a single centralized reinforcement‑learning (RL) controller, or multiple learners that synchronize their updates—suffer from scalability bottlenecks and “straggler” effects: as the network grows, the state‑action space explodes and slower agents hold up the entire learning pipeline.

To overcome these limitations, the authors propose an Asynchronous Multi‑Agent Reinforcement Learning (AMARL) framework. The key idea is to allocate one independent Proximal Policy Optimization (PPO) agent to each service class. Each agent observes only the flows belonging to its service, runs local episodic simulations, and, upon completing a routing decision, emits a resource delta (Δ) that reflects the bandwidth consumed on each traversed link and the compute load on the virtual network functions (VNFs) used. This delta is applied to a shared global environment (E★) that maintains the current utilization of all links, compute nodes, and a time‑of‑day index. The commit operation checks capacity and latency constraints; if a violation would occur, the commit is aborted and the episode is marked as rejected. In this way, agents never directly synchronize; coordination emerges implicitly through contention for shared resources—a mechanism the authors call “coordination‑by‑state.”

The asynchronous design yields two major benefits. First, agents operate on their own cadences (e.g., 10‑100 ms), mirroring the heterogeneous timing of real‑world xApps in an O‑RAN architecture, and thus avoid idle waiting caused by slower peers. Second, rollout (environment interaction) and learning (policy update) are decoupled, dramatically increasing sample throughput. The implementation leverages Ray RLlib for the PPO trainers and Ray Actors for the global environment, providing a scalable, fault‑tolerant execution platform.

Problem formulation is cast as a multi‑commodity flow optimization on a directed graph G(V,E). Each link e has capacity Cₑ and propagation delay tₑ; each compute node v has a usable compute budget κᵥ. Services are defined by ordered Service Function Chains (SFCs) fₛ, and each flow p carries source, destination, bandwidth bₚ, and a latency budget τₚ. Decision variables include admission (accₚ), per‑stage placement (aᵥ,ₖ,ₚ), and link usage (yₑ,ₚ,ₖ). Constraints enforce single execution of each SFC stage, link capacity, and compute capacity, while the reward balances acceptance, latency compliance, and penalty for violations.

Related work is surveyed across three axes: (1) MARL for networking, which typically uses synchronous updates and suffers from non‑stationarity; (2) Asynchronous RL (A3C, IMPALA, Ape‑X), which shows wall‑time speedups in single‑agent settings but has rarely been applied to cooperative multi‑agent routing; (3) RL for O‑RAN, where most efforts focus on slicing or scheduling rather than routing. The authors argue that no prior work combines full asynchrony, service‑specific agents, and a shared network state for routing.

Experimental evaluation uses a realistic 24‑hour traffic trace from Montreal, mapped onto a 24‑node O‑RAN‑like topology. Six services are modeled, each with its own PPO agent (S = 6). The baseline is a single‑agent PPO (named SARL) that handles all flows jointly. Both approaches share the same reward function and network model. Results show that AMARL achieves comparable or slightly better Grade of Service (acceptance rate) and average end‑to‑end latency than SARL, while reducing training wall‑clock time by roughly 30 % and inference time by 15 %. Moreover, under sudden demand spikes (e.g., rush‑hour peaks), the service‑specialized agents adapt more gracefully, maintaining higher acceptance and lower latency, demonstrating improved robustness to demand shifts.

Limitations and future directions are candidly discussed. The current design assumes a fixed number of services; scaling to dozens of service classes would increase the number of agents and potentially the coordination overhead. The lock‑based commit on the shared environment could become a bottleneck in very large topologies, suggesting the need for more sophisticated distributed transaction mechanisms. Future work includes (i) dynamic scaling of agents, (ii) meta‑learning to quickly accommodate new services, (iii) integration of dynamic VNF placement with routing, and (iv) deployment on an actual O‑RAN testbed to validate real‑world performance.

In summary, the paper provides a practical, scalable, and robust solution for 5G routing by marrying asynchronous execution with service‑specific MARL. The empirical evidence confirms that decoupling agents and relying on shared state for implicit coordination can deliver wall‑time speedups without sacrificing QoS, positioning AMARL as a promising building block for next‑generation network automation and O‑RAN xApp ecosystems.