Low-Complexity EP-Net–Based Signal Detection for High-Mobility MIMO-OTFS Systems Over Rayleigh and Rician Channels

Signal detection in delay–Doppler domain multiple-input multiple-output orthogonal time frequency space (MIMO-OTFS) systems remains challenging due to severe inter-symbol interference, high computational complexity, and pronounced performance degradation under channel estimation errors, especially in high-mobility scenarios. To address these issues, this paper proposes a low-complexity, model-driven Expectation Propagation Network (EP-Net)–based signal detection framework for MIMO-OTFS systems operating over Rayleigh and Rician fading channels. The proposed EP-Net unfolds the classical expectation propagation algorithm into a finite-depth neural network with reduced iterations, shared learnable parameters, and avoidance of repeated matrix inversions, thereby significantly lowering computational complexity while preserving probabilistic interpretability. Extensive MATLAB simulations are conducted to evaluate both bit error rate (BER) and power spectral density (PSD) performance under practical channel estimation errors. Numerical results demonstrate that, at a target BER of , the proposed EP-Net achieves 8–12 dB signal-to-noise ratio gain over conventional linear, iterative, and learning-based detectors, depending on the fading environment and estimation accuracy. Moreover, PSD analysis shows that EP-Net provides up to 50 dB reduction in out-of-band emissions, indicating superior spectral confinement and interference suppression. These results confirm that the proposed EP-Net detector offers a robust, spectrally efficient, and computationally viable solution for high-mobility and 6G-oriented MIMO-OTFS wireless communication systems.