ReaCritic: Reasoning Transformer-based DRL Critic-model Scaling For Wireless Networks

Heterogeneous Networks (HetNets) pose critical challenges for intelligent management due to the diverse user requirements and time-varying wireless conditions. These factors introduce significant decision complexity, which limits the adaptability of existing Deep Reinforcement Learning (DRL) methods. In many DRL algorithms, especially those involving value-based or actor-critic structures, the critic component plays a key role in guiding policy learning by estimating value functions. However, conventional critic models often use shallow architectures that map observations directly to scalar estimates, limiting their ability to handle multi-task complexity. In contrast, recent progress in inference-time scaling of Large Language Models (LLMs) has shown that generating intermediate reasoning steps can significantly improve decision quality. Motivated by this, we propose ReaCritic, a reasoning transformer-based critic-model scaling scheme that brings reasoning-like ability into DRL. ReaCritic performs horizontal reasoning over parallel state-action inputs and vertical reasoning through deep transformer stacks. It is compatible with a broad range of value-based and actor-critic DRL algorithms and enhances generalization in dynamic wireless environments. Extensive experiments demonstrate that ReaCritic improves convergence speed and final performance across various HetNet settings and standard OpenAI Gym control tasks. The code of ReaCritic is available at https://github.com/NICE-HKU/ReaCritic.

💡 Research Summary

The paper tackles the scalability and generalization challenges of deep reinforcement learning (DRL) when applied to large‑scale heterogeneous networks (HetNets), where the state‑action space is high‑dimensional, highly coupled, and non‑stationary due to user mobility, varying traffic, and multi‑tier interference. Existing DRL solutions typically employ shallow multi‑layer perceptron (MLP) critics that map flattened observations directly to scalar value estimates. Such critics lack the capacity to capture latent dependencies among state variables and to reason over multi‑objective QoS metrics (latency, throughput, energy efficiency), leading to unstable training, slow convergence, and poor performance in unseen or rapidly changing scenarios.

Inspired by the reasoning capabilities of large language models (LLMs), the authors argue that these capabilities stem from the transformer architecture—self‑attention and hierarchical composition—rather than from language data per se. To avoid the drawbacks of plugging external LLMs (non‑determinism, high computational cost, lack of convergence guarantees), they embed a transformer‑based reasoning module directly into the critic, naming the resulting framework ReaCritic.

ReaCritic introduces two complementary reasoning dimensions:

1. Horizontal Reasoning (HRea) – The current state‑action pair is tokenized into a sequence of parallel tokens, each representing a sub‑component of the high‑dimensional input (e.g., per‑user channel state, per‑resource allocation). Multi‑head self‑attention processes all tokens simultaneously, allowing the critic to explore a broad set of correlations across the entire input space rather than compressing everything into a single vector.

2. Vertical Reasoning (VRea) – The token sequence is passed through a stack of transformer blocks (the “vertical” depth). Each block abstracts the representation further, enabling the model to capture long‑range dependencies, temporal dynamics, and non‑stationary patterns in the wireless environment. The number of vertical layers (V) and horizontal tokens (H) can be scaled adaptively to match task complexity, providing a form of inference‑time scaling analogous to LLM prompting.

By placing this reasoning engine in the critic, the policy (actor) network can remain unchanged, preserving the original action‑selection mechanism while receiving richer, more accurate gradient signals. The design is compatible with a wide range of value‑based and actor‑critic algorithms (DDQN, DDPG, SAC, A2C, etc.). The authors also adopt a centralized training architecture, which offloads the heavy transformer computation to a cloud or edge server, avoiding excessive load on resource‑constrained devices.

Experimental Evaluation

• HetNet Simulations: A realistic multi‑user, multi‑tier HetNet simulator is built, incorporating Rician fading, stochastic traffic bursts, and heterogeneous QoS requirements. Baselines include standard MLP critics, an LLM‑aided critic, and several state‑of‑the‑art DRL algorithms. ReaCritic consistently achieves 15–20 % faster convergence and a 10 %+ improvement in final cumulative reward across varying numbers of users (up to 200) and dynamic channel conditions. Ablation studies show that both horizontal and vertical reasoning contribute additively to performance gains.
• OpenAI Gym Continuous Control: ReaCritic is plugged into DDPG and SAC for classic control tasks (Pendulum, HalfCheetah, Walker2d). Results demonstrate higher sample efficiency (fewer episodes to reach a target score) and superior asymptotic performance (5–12 % higher returns) compared with the original critics.

Strengths

• Novel adaptation of transformer reasoning to the critic, directly addressing the representation bottleneck in DRL for wireless networks.
• Clear separation of horizontal (breadth) and vertical (depth) scaling, enabling flexible resource allocation.
• Extensive empirical validation on both domain‑specific (HetNet) and generic (Gym) benchmarks.
• Compatibility with existing DRL pipelines, requiring only a critic swap.

Limitations & Open Issues

• Transformer‑based critics increase memory and compute requirements; while the authors propose configurable H and V, real‑time edge deployment would still benefit from model compression (pruning, quantization, knowledge distillation).
• The tokenization strategy for horizontal reasoning is heuristic; a principled method for selecting which state‑action components become tokens could further improve efficiency.
• The paper focuses on a centralized training setting; extending ReaCritic to fully distributed multi‑agent scenarios remains an open research direction.

Conclusion & Future Work
ReaCritic demonstrates that embedding transformer‑style reasoning into the DRL critic dramatically enhances scalability, convergence speed, and robustness in highly dynamic wireless environments. Future research may explore joint transformer designs for both actor and critic, adaptive token selection mechanisms, and on‑device lightweight implementations to bring the benefits of reasoning‑augmented DRL to real 5G/6G deployments.