Une approche modulaire probabiliste pour le routage `a Qualite de Service integree

Due to emerging real-time and multimedia applications, efficient routing of information packets in dynamically changing communication network requires that as the load levels, traffic patterns and topology of the network change, the routing policy also adapts. We focused in this paper on QoS based routing by developing a neuro-dynamic programming to construct dynamic state dependent routing policies. We propose an approach based on adaptive algorithm for packet routing using reinforcement learning which optimizes two criteria: cumulative cost path and end-to-end delay. Numerical results obtained with OPNET simulator for different packet interarrival times statistical distributions with different levels of traffic’s load show that the proposed approach gives better results compared to standard optimal path routing algorithms.

💡 Research Summary

The paper addresses the challenge of routing in modern communication networks where traffic patterns, load levels, and topology can change rapidly due to real‑time and multimedia applications. Traditional static routing schemes are ill‑suited for such environments because they cannot react quickly to dynamic conditions without over‑provisioning resources. To overcome these limitations, the authors propose a modular, probabilistic routing framework that combines static QoS metrics with dynamically measured network parameters, and they embed a reinforcement‑learning (RL) component based on Q‑Learning to adapt routing decisions in real time.

The network is modeled as a directed graph G = (X, U) where nodes represent routers or switches and edges represent communication links. Each link (u, v) carries an m‑dimensional vector W(u, v) of QoS attributes (e.g., bandwidth, delay, loss rate). The routing problem is to select K candidate paths between a source and a destination that jointly satisfy static constraints (bandwidth, hop count, baseline delay, error rate) and dynamic constraints (current queue length, measured delay, jitter, packet loss). Because optimizing more than one non‑correlated objective is NP‑complete, the authors restrict the optimization to two primary criteria: cumulative link cost and end‑to‑end delay.

The proposed algorithm operates in two stages. First, a static‑cost function f(bandwidth, hops, delay, error) is defined and used to compute the K shortest paths using a variant of Eppstein’s K‑Shortest‑Path algorithm. This yields a set of candidate routes that are promising from a static perspective. Second, the RL module assigns a Q‑value to each candidate path and updates it whenever a packet traverses the path. The dynamic cost function f′(availability, loss, measured delay, jitter, measured bandwidth) feeds into the Q‑value update, thus reflecting the current state of the network.

Two probabilistic exploration strategies are introduced. The first, called KSPQR (K‑Shortest‑Path Q‑Routing), allocates a maximum probability Pmax to the path with the highest current Q‑value and distributes the remaining probability uniformly among the other K‑1 paths: each receives (1‑Pmax)/K. The second, KOQRA (K‑Optimal‑Q‑Routing‑Adaptive), draws inspiration from Ant Colony Optimization. It computes an adaptive probability for each path based on two parameters: the estimated end‑to‑end delay and the waiting time at the next router (queue length). By adjusting probabilities adaptively, KOQRA discourages sending packets through interfaces whose downstream queues are saturated, even if those interfaces belong to the shortest‑delay path.

The learning process follows a standard Q‑Learning rule with a reward that penalizes high delay and high loss, and an ε‑greedy‑like exploration policy to balance exploitation of the best path with occasional probing of alternatives. The authors limit Q‑value updates to the K selected paths, thereby keeping the computational overhead modest.

Simulation experiments were conducted on the OPNET platform using the Japanese NTTnet topology. Traffic arrivals were modeled as a Poisson process, and three load scenarios were examined: low load, high load, and traffic spikes. The proposed algorithms (KSPQR and KOQRA) were compared against conventional Shortest Path First (SPF) and Standard Optimal Multi‑Path Routing (SOMR). Results show that under low load, SPF and SOMR slightly outperform the learning‑based methods because the latter generate additional control packets that marginally increase delay (≈15 %). However, under high load and especially during traffic spikes, both KSPQR and KOQRA achieve significant improvements: average packet delivery time is reduced by roughly 12 % (KSPQR) and 15 % (KOQRA) relative to SPF/SOMR. The performance gain stems from the algorithms’ ability to react quickly to congestion by dynamically adjusting path probabilities, thereby redistributing traffic away from overloaded links. The control‑packet overhead introduced by the learning process remains negligible compared with the total traffic volume, supporting the practicality of the approach.

Key contributions of the paper include: (1) a unified cost model that fuses static and dynamic QoS metrics; (2) a two‑stage routing framework that first selects K promising static paths and then refines their usage through Q‑Learning; (3) probabilistic exploration mechanisms (KSPQR and KOQRA) that balance learning speed with network stability; and (4) an extensive simulation‑based validation on a realistic network topology.

The authors acknowledge several limitations. The performance is sensitive to the choice of learning parameters (Pmax, ε, learning rate), which may need retuning for different network conditions. Moreover, the computational complexity grows with the number of candidate paths K (O(kN log(kN) + k² mM)), potentially limiting scalability in very large networks. Future work is proposed to extend the reinforcement signal to incorporate additional QoS dimensions such as bandwidth guarantees and packet loss thresholds, and to explore distributed learning architectures that can scale to larger topologies while preserving low overhead.

In conclusion, the modular probabilistic approach presented demonstrates that integrating reinforcement learning with multi‑path routing can substantially improve end‑to‑end delay and overall QoS in highly dynamic network environments, especially under heavy traffic conditions where traditional static routing struggles to adapt.