Distributive Stochastic Learning for Delay-Optimal OFDMA Power and Subband Allocation

In this paper, we consider the distributive queue-aware power and subband allocation design for a delay-optimal OFDMA uplink system with one base station, $K$ users and $N_F$ independent subbands. Each mobile has an uplink queue with heterogeneous packet arrivals and delay requirements. We model the problem as an infinite horizon average reward Markov Decision Problem (MDP) where the control actions are functions of the instantaneous Channel State Information (CSI) as well as the joint Queue State Information (QSI). To address the distributive requirement and the issue of exponential memory requirement and computational complexity, we approximate the subband allocation Q-factor by the sum of the per-user subband allocation Q-factor and derive a distributive online stochastic learning algorithm to estimate the per-user Q-factor and the Lagrange multipliers (LM) simultaneously and determine the control actions using an auction mechanism. We show that under the proposed auction mechanism, the distributive online learning converges almost surely (with probability 1). For illustration, we apply the proposed distributive stochastic learning framework to an application example with exponential packet size distribution. We show that the delay-optimal power control has the {\em multi-level water-filling} structure where the CSI determines the instantaneous power allocation and the QSI determines the water-level. The proposed algorithm has linear signaling overhead and computational complexity $\mathcal O(KN)$, which is desirable from an implementation perspective.

💡 Research Summary

The paper addresses the problem of jointly allocating transmit power and sub‑bands in an uplink OFDMA system so as to minimize long‑term average packet delay while respecting power and throughput constraints. The system consists of a single base station, K mobile users, and N_F orthogonal sub‑bands. Each user maintains a separate uplink queue with heterogeneous packet arrival processes and distinct delay requirements. By modeling the control problem as an infinite‑horizon average‑reward Markov Decision Process (MDP), the authors capture the coupling between instantaneous channel state information (CSI) and the joint queue state information (QSI).

A direct solution of the MDP is infeasible because the state‑action space grows exponentially with K·N_F, leading to prohibitive memory and computational demands. To overcome this, the authors propose a structural approximation: the global Q‑factor is expressed as the sum of per‑user sub‑band Q‑factors, i.e., Q(s,a)≈∑_{k=1}^K Q_k(s_k,a_k). This decomposition enables each user to learn its own Q‑factor using only locally observed CSI, QSI, and its own actions, thereby eliminating the need for a centralized value table.

The learning algorithm operates online and consists of two coupled stochastic approximation loops. In the fast time‑scale, each user updates its Q‑factor with step size α(t) based on the instantaneous reward (a weighted sum of delay cost and power cost) and the estimated future value. In the slow time‑scale, Lagrange multipliers associated with the average power budget and minimum throughput constraints are updated with step size β(t). The two step‑size sequences satisfy the standard diminishing‑step conditions (∑α(t)=∞, ∑α(t)^2<∞, similarly for β(t)), guaranteeing almost‑sure convergence of both the Q‑factors and the multipliers.

Resource allocation is performed through an auction mechanism. At each scheduling interval, every user computes a bid for each sub‑band using its current Q‑factor and the latest multiplier values; the bid reflects the expected reduction in delay versus the incurred power cost. The base station then assigns each sub‑band to the user with the highest bid. This procedure is fully distributed—users need only exchange their bids—resulting in a linear signaling overhead O(K·N) and a computational complexity O(K·N).

The authors provide a rigorous convergence proof. The Q‑factor update is shown to be an unbiased stochastic approximation of the Bellman equation, and under the diminishing‑step conditions it converges almost surely to a fixed point. The multiplier update is analyzed via a Lyapunov stability argument, establishing its almost‑sure convergence to the set of saddle points that satisfy the power and throughput constraints. Consequently, the overall algorithm converges with probability one to a policy that is asymptotically optimal for the original MDP.

To illustrate the framework, the paper examines a scenario where packet sizes follow an exponential distribution. In this setting, the optimal power control exhibits a “multi‑level water‑filling” structure: the water level is not only a function of CSI (as in classic water‑filling) but also of QSI. When a user’s queue builds up, its water level rises, prompting the algorithm to allocate more power to that user; conversely, a short queue lowers the water level, conserving power. This adaptive behavior directly reduces queueing delay.

Simulation results confirm that the proposed distributed stochastic learning algorithm achieves substantially lower average packet delay compared with conventional queue‑agnostic power control and with centralized optimal solutions that are computationally prohibitive. The algorithm respects the average power constraint, scales linearly with the number of users and sub‑bands, and adapts online to time‑varying traffic and channel conditions.

In summary, the paper contributes a novel distributed learning architecture for delay‑optimal OFDMA resource allocation. By decomposing the Q‑factor, employing simultaneous stochastic learning of value functions and Lagrange multipliers, and using an auction‑based scheduling rule, the authors reconcile the conflicting goals of optimality, scalability, and low signaling overhead. The work opens avenues for extensions to multi‑cell coordination, non‑exponential traffic models, and deep‑reinforcement‑learning based value approximations.