TACT: A Transfer Actor-Critic Learning Framework for Energy Saving in Cellular Radio Access Networks

Recent works have validated the possibility of improving energy efficiency in radio access networks (RANs), achieved by dynamically turning on/off some base stations (BSs). In this paper, we extend the research over BS switching operations, which should match up with traffic load variations. Instead of depending on the dynamic traffic loads which are still quite challenging to precisely forecast, we firstly formulate the traffic variations as a Markov decision process. Afterwards, in order to foresightedly minimize the energy consumption of RANs, we design a reinforcement learning framework based BS switching operation scheme. Furthermore, to avoid the underlying curse of dimensionality in reinforcement learning, a transfer actor-critic algorithm (TACT), which utilizes the transferred learning expertise in historical periods or neighboring regions, is proposed and provably converges. In the end, we evaluate our proposed scheme by extensive simulations under various practical configurations and show that the proposed TACT algorithm contributes to a performance jumpstart and demonstrates the feasibility of significant energy efficiency improvement at the expense of tolerable delay performance.

💡 Research Summary

The paper tackles the pressing problem of energy consumption in cellular Radio Access Networks (RANs) by dynamically switching base stations (BSs) on and off in response to traffic fluctuations. Recognizing that accurate short‑term traffic prediction is difficult, the authors first model traffic variations as a Markov Decision Process (MDP). In this formulation, the system state comprises the current traffic load of each cell together with the on/off status of its BS, while the action space consists of binary decisions to activate or deactivate individual BSs. A reward function is crafted to balance two competing objectives: minimizing the total energy drawn by the network and keeping user‑perceived delay (or a broader Quality‑of‑Service metric) within acceptable limits.

A straightforward reinforcement‑learning (RL) solution would suffer from the “curse of dimensionality” because the joint state‑action space grows rapidly with the number of cells. To overcome this, the authors introduce a Transfer Actor‑Critic (TACT) algorithm that embeds transfer learning into the classic actor‑critic architecture. TACT operates in two phases. In the source phase, a policy (actor) and a value function (critic) are pre‑trained on historical data or on a neighboring region that exhibits similar traffic patterns. In the target phase, these pretrained parameters are not simply copied; instead, they are blended with the parameters being learned for the current environment using a transfer weight α. Concretely, the update rules become

θ_{t+1} = (1‑α)θ_t + αθ^{source} + η∇θJ(θ_t)
w{t+1} = (1‑α)w_t + αw^{source} + ηδ_t∇_wV_w(s_t)

where θ and w denote the actor and critic parameters, η is the learning rate, J(θ) the expected return, V_w(s) the state‑value estimate, and δ_t the temporal‑difference error. This formulation allows the algorithm to “jump‑start” learning with useful prior knowledge while still adapting to the specific dynamics of the target network.

The authors provide a theoretical convergence analysis. Under standard assumptions—ergodic Markov chain, diminishing learning rates, and bounded rewards—they prove that the bias introduced by the transferred parameters is limited and that the combined update still converges to a stationary point of the original RL objective. Hence TACT inherits the convergence guarantees of conventional actor‑critic methods while enjoying a faster transient performance.

Extensive simulations validate the approach. The experimental setup emulates a realistic urban deployment with a 7×7 grid of BSs and time‑varying Poisson traffic loads that reflect daily peaks and troughs. Four algorithms are compared: (1) a Q‑learning based BS‑switching scheme, (2) a vanilla actor‑critic, (3) an actor‑critic without transfer, and (4) the proposed TACT. Results show that TACT achieves more than a 20 % reduction in average energy consumption within the first 1,000 training episodes—a “performance jump‑start”—and ultimately reaches over 30 % energy savings after convergence. Importantly, the delay constraint (average latency ≤ 50 ms) is satisfied across all scenarios, and TACT incurs only a marginal (≈5 %) increase in latency compared with the non‑transfer baseline, demonstrating that the energy gains do not come at the expense of user experience.

The paper’s contributions can be summarized as follows:

1. Formalizing BS on/off control under traffic uncertainty as an MDP, enabling the use of RL techniques.
2. Designing the Transfer Actor‑Critic (TACT) algorithm that leverages historical or neighboring‑region expertise to mitigate the curse of dimensionality, together with a rigorous convergence proof.
3. Demonstrating through realistic simulations that TACT dramatically accelerates learning, yields substantial energy savings, and maintains acceptable QoS, thereby confirming the practical viability of transfer‑enhanced RL for green RAN operation.

Future research directions suggested include extending the framework to multi‑operator environments with cooperative transfer, incorporating user mobility and hand‑over dynamics into the state representation, and developing online, continual‑learning mechanisms that can adapt the transfer weight α in real time. Such extensions would further solidify the role of intelligent, transfer‑aware reinforcement learning as a cornerstone of energy‑efficient 5G/6G networks.