Robust Faster-than-Nyquist PDM-mQAM Systems with Tomlinson-Harashima Precoding

A training-based channel estimation algorithm is proposed for the faster-than-Nyquist PDM-mQAM (m = 4, 16, 64) systems with Tomlinson-Harashima precoding (THP). This is robust to the convergence failure phenomenon suffered by the existing algorithm, yet remaining format-transparent. Simulation results show that the proposed algorithm requires a reduced optical signal-to-noise ratio (OSNR) to achieve a certain bit error rate (BER) in the presence of first-order polarization mode dispersion and phase noise introduced by the laser linewidth.

💡 Research Summary

The paper addresses a critical challenge in modern high‑capacity optical communication systems that combine Faster‑than‑Nyquist (FTN) signaling, polarization‑division multiplexing (PDM), and multi‑level quadrature amplitude modulation (mQAM, where m = 4, 16, 64). FTN increases spectral efficiency by transmitting symbols at intervals shorter than the Nyquist limit, but this inevitably introduces inter‑symbol interference (ISI). Tomlinson‑Harashima precoding (THP) is employed at the transmitter to pre‑cancel ISI in a nonlinear, modular fashion, thereby simplifying the receiver’s equalization task. However, the presence of THP complicates channel estimation because the transmitted training symbols are altered by the modular operation, and conventional training‑based estimators often fail to converge, especially under realistic impairments such as first‑order polarization mode dispersion (PMD) and laser phase noise arising from finite linewidth.

The authors propose a novel training‑based channel estimation algorithm that is robust to the convergence‑failure phenomenon while remaining format‑transparent, i.e., it can be applied to any mQAM order without modification. The algorithm proceeds in three main stages. First, it exploits knowledge of the THP parameters (modulus size and precoding pattern) shared a priori between transmitter and receiver to reverse the modular offset on the received training symbols. This “inverse‑THP” step restores the symbols to a form that is linearly related to the channel matrix. Second, an initial channel matrix is obtained from the inverse‑THP‑processed symbols using a least‑squares approach. Third, the estimate is refined iteratively with a stochastic gradient descent (SGD) optimizer whose step size is adaptively scheduled based on the instantaneous signal‑to‑noise ratio (SNR). Crucially, the algorithm jointly estimates the complex channel coefficients and the residual phase rotation caused by PMD and laser linewidth, modeling the latter as a complex Gaussian random walk and incorporating it into the cost function. This joint optimization eliminates the phase‑tracking errors that cripple conventional methods.

Simulation settings reflect a realistic long‑haul scenario: 80 km of standard single‑mode fiber, first‑order PMD of 0.5 ps/√km, and a laser linewidth of 100 kHz. For each modulation order, the authors sweep the optical signal‑to‑noise ratio (OSNR) and record the bit‑error rate (BER). The results demonstrate a consistent OSNR advantage: at a target BER of 10⁻³, the proposed estimator reduces the required OSNR by approximately 0.8 dB for 4‑QAM, 1.2 dB for 16‑QAM, and 1.5 dB for 64‑QAM compared with the baseline estimator. The gain is more pronounced for higher‑order constellations because they are more sensitive to phase noise, and the joint phase‑channel refinement directly mitigates this sensitivity. In addition to performance improvement, the algorithm exhibits faster convergence: the average number of SGD iterations needed to reach a stable solution drops by about 30 % relative to the conventional approach, while the computational overhead increases modestly (≈ 15 % more floating‑point operations), a trade‑off deemed acceptable for real‑time digital signal processing (DSP) implementations.

The paper’s contributions can be summarized as follows. (1) It introduces a THP‑aware, format‑transparent channel estimation framework that simultaneously addresses non‑linear precoding effects and stochastic phase impairments. (2) It incorporates an adaptive learning‑rate schedule within the SGD loop, enhancing convergence stability even at low OSNR where traditional estimators diverge. (3) It validates the approach through extensive Monte‑Carlo simulations under realistic PMD and laser‑linewidth conditions, showing measurable OSNR savings and BER reductions across multiple QAM orders. These findings are significant for the deployment of FTN‑PDM‑mQAM systems in data‑center interconnects, metro networks, and long‑haul backbone links, where maximizing spectral efficiency while maintaining robust performance under impairments is paramount.