Deep Learning-based OTFS Channel Estimation and Symbol Detection with Plug-and-Play Framework

Orthogonal Time Frequency Space (OTFS) modulation has recently attracted significant interest due to its potential for enabling reliable communication in high-mobility environments. However, the effectiveness of OTFS receivers relies on the inherent characteristic of the Delay-Doppler (DD) domain channel, where the sparsity of the discretized channel varies across different communication scenarios. For instance, the fractional Doppler effect reduces the inherent channel sparsity, which consequently degrades channel estimation accuracy and increases the complexity of symbol detection. Traditional algorithms relying on fixed sparsity priors often require manual design, while purely data-driven deep learning (DL) methods typically struggle to generalize across diverse channel conditions. To address these challenges, we propose a novel unsupervised DL-based plug-and-play (PnP) framework that provides a flexible solution for OTFS receiver design. The proposed framework can be applied to both channel estimation and symbol detection, jointly leveraging the flexibility of optimization-based methods and the powerful generalization capability of data-driven models. Specifically, a lightweight encoder-decoder network (EDN) is incorporated as an implicit channel prior for channel estimation, enabling robust performance across varying levels of channel sparsity. Furthermore, for symbol detection, we realize the PnP framework with a time-domain matrix inversion for model-based equalization, followed by a small multi-layer perceptron (MLP) pre-trained for specific constellations, thereby achieving low complexity and enabling flexible adaptation to various modulation formats. Finally, numerical results demonstrate the effectiveness and robustness of the algorithm.

💡 Research Summary

This paper addresses two fundamental challenges in Orthogonal Time Frequency Space (OTFS) communications—accurate channel estimation and reliable symbol detection—by introducing a unified, unsupervised deep learning (DL) plug‑and‑play (PnP) framework. OTFS operates in the delay‑Doppler (DD) domain, where the channel is typically sparse and quasi‑static, which makes it attractive for high‑mobility scenarios such as LEO satellites, UAVs, and vehicular networks. However, practical channels often exhibit fractional Doppler shifts that smear the sparsity, degrading the performance of conventional estimators (LS, LMMSE) and sparsity‑aware algorithms (OMP, SBL). Likewise, message‑passing detectors become computationally prohibitive when sparsity diminishes and are highly sensitive to channel estimation errors.

The authors formulate both tasks as linear inverse problems and apply variable‑splitting (e.g., ADMM) to separate a model‑based data‑fidelity subproblem from a learning‑based denoising subproblem. The denoiser acts as an implicit prior that can be swapped without altering the physical consistency step, which is the essence of the PnP paradigm.

Channel Estimation (PnP‑CE).
A lightweight encoder‑decoder network (EDN) is employed as the prior. The EDN receives a coarse LS estimate and, through multi‑scale convolutions and skip connections, learns to reconstruct the true DD‑domain channel, implicitly handling sparsity variations, leakage, and fractional Doppler effects. Training is unsupervised: the loss combines a data‑consistency term (enforcing the OTFS input‑output relation) with an L2 regularization on the reconstructed channel. Training data span a wide range of SNRs, Doppler spreads, and sparsity levels, enabling the EDN to generalize without retraining for new channel conditions.

Symbol Detection (PnP‑SD).
The detection pipeline first performs an efficient time‑domain matrix inversion exploiting the block‑diagonal structure of the OTFS modulation matrix, drastically reducing computational load to O(N M). The resulting soft estimate is then passed through a small, constellation‑aware multi‑layer perceptron (MLP). The MLP is trained separately for each modulation format (e.g., QPSK, 16‑QAM) but remains shallow, allowing real‑time execution. By embedding constellation knowledge, the MLP denoiser mitigates the impact of imperfect channel state information and improves bit‑error‑rate (BER) performance.

Complexity and Convergence.
Compared with traditional OMP‑based detectors, the proposed PnP‑SD reduces arithmetic operations by more than 30 % while achieving comparable or better BER. The PnP framework inherits convergence guarantees from existing imaging literature under mild Lipschitz conditions, ensuring stable iterative updates despite the non‑convexity introduced by the neural priors.

Simulation Results.
Extensive Monte‑Carlo simulations cover integer and fractional Doppler scenarios, varying numbers of multipath components (P = 4–8), and SNRs from 0 to 30 dB. For channel estimation, PnP‑CE attains mean‑square‑error (MSE) improvements of 3–5 dB over LS/LMMSE and outperforms a recent deep residual network, especially when fractional Doppler spreads are large. For detection, PnP‑SD achieves BER gains of 2–4 dB relative to message‑passing, ViterbiNet, and SBL‑based detectors, while maintaining low computational complexity. Importantly, performance degradation remains modest as channel sparsity diminishes, demonstrating robustness to the very effect that hampers many sparsity‑based methods.

Contributions.

1. Unified PnP formulation that integrates model‑based physics with flexible DL priors for both estimation and detection.
2. Unsupervised training of a compact encoder‑decoder prior that adapts to varying sparsity and fractional Doppler without retraining.
3. Hybrid‑domain detector combining efficient time‑domain inversion and a constellation‑aware MLP, yielding low complexity and modulation‑agnostic operation.
4. Comprehensive evaluation showing superior accuracy and robustness compared with state‑of‑the‑art algorithms across diverse channel conditions.

Future Directions.
The authors suggest extending the framework to massive MIMO OTFS, real‑time hardware implementation, and online adaptation of the neural priors to further enhance practicality.

In summary, the paper presents a novel, flexible, and computationally efficient solution for OTFS receivers, leveraging the strengths of both optimization‑based signal processing and deep learning within a plug‑and‑play architecture.