A Reinforcement Learning Based Universal Sequence Design for Polar Codes

To advance Polar code design for 6G applications, we develop a reinforcement learning-based universal sequence design framework that is extensible and adaptable to diverse channel conditions and decoding strategies. Crucially, our method scales to code lengths up to $2048$, making it suitable for use in standardization. Across all $(N,K)$ configurations supported in 5G, our approach achieves competitive performance relative to the NR sequence adopted in 5G and yields up to a 0.2 dB gain over the beta-expansion baseline at $N=2048$. We further highlight the key elements that enabled learning at scale: (i) incorporation of physical law constrained learning grounded in the universal partial order property of Polar codes, (ii) exploitation of the weak long term influence of decisions to limit lookahead evaluation, and (iii) joint multi-configuration optimization to increase learning efficiency.

💡 Research Summary

The paper addresses the need for a universal polar‑code reliability ordering that can be used across all block lengths and rates envisioned for 6G, extending the current 5G NR practice which is limited to N ≤ 1024. The authors formulate the problem as finding a single stochastic policy that, when sampled, yields an absolute ordering of synthetic channels whose first K indices are the most reliable for any (N, K) pair. This ordering must outperform the beta‑expansion baseline in block error rate (BLER) while being scalable to N = 2048.

To achieve this, the authors combine three key ideas. First, they embed the Universal Partial Order (UPO) – a set of deterministic reliability relations that hold for successive‑cancellation (SC) decoding – as hard constraints in the action space. By enforcing UPO, the combinatorial search space for N = 2048 is reduced from roughly 10^5894 to 10^2582 possible sequences, making the problem tractable. Second, they adopt Proximal Policy Optimization (PPO) as the deep‑RL backbone, which eliminates the need for experience replay and provides stable on‑policy updates. Third, they introduce a multi‑configuration joint optimization framework: the policy is trained simultaneously on a range of (N, K) configurations, allowing knowledge learned on small block lengths to be transferred to larger ones, dramatically improving sample efficiency.

The method also incorporates several practical refinements. “Lower‑N embedding” guarantees that the optimal ordering found for a smaller block length is preserved when the search proceeds to the next power‑of‑two length, thereby protecting performance at short lengths. Because strict embedding can hurt large‑N performance, the authors relax the ordering of the first K_min bits (which have negligible impact on smaller codes) and extend the action space with one‑hop UPO violations (the “UPO+” rule) to capture candidates that are beneficial under successive‑cancellation‑list (SCL) decoding. A Monte‑Carlo‑Tree‑Search‑inspired limited look‑ahead is used to evaluate rewards without exhaustively simulating every possible K, exploiting the fact that early decisions affect all larger K.

Experimental results cover all block lengths supported by 5G NR (N = 32, 64, 128, 256, 512, 1024, 2048) and a wide range of rates. The learned universal sequence matches or slightly exceeds the NR reference sequence across the board and delivers up to a 0.2 dB gain over the beta‑expansion baseline at N = 2048. Training converges within a few tens of hours on a single GPU, representing a 5× speed‑up compared with prior deep‑RL approaches that were limited to much shorter lengths.

In summary, the work demonstrates that physics‑informed constraints (UPO), modern on‑policy reinforcement learning (PPO), and joint multi‑configuration training can together produce a scalable, high‑performance universal polar‑code sequence suitable for future 6G standardization. The authors also release their code publicly, paving the way for further research on meta‑learning, adaptive decoding, and hardware‑friendly implementations.