Coupled-Interference Modeled FTN Signaling Over Doubly Selective Fading Channels: Joint Subpath Recovery and Iterative Detection

Faster-than-Nyquist (FTN) technique promises higher capacity and spectral efficiency for wireless communications. However, existing FTN studies over doubly-selective fading (DSF) channels separate channel-induced inter-symbol interference (channel-ISI) and FTN-induced ISI (FTN-ISI) to simplify cancellation. In practical DSF scenarios, the inherent coupling between FTN-ISI and channel-ISI causes significant performance degradation in conventional detection algorithms. To address this limitation, we first derive a practical FTN transmission model over DSF channels and construct the corresponding coupled interference matrix. Considering that data detection relies on efficient channel estimation, we propose a channel estimation algorithm with joint recovery of resolvable subpath parameters. This algorithm decomposes propagation paths into resolvable subpaths with independent delay-Doppler characteristics, achieving enhanced estimation accuracy through joint gain-phase optimization. Finally, building on the derived transceiver model and coupled interference matrix, we propose a whitening-enhanced orthogonal approximate message passing (WE-OAMP) algorithm that suppresses coupled interference through iterative linear-nonlinear estimation while maintaining spectral compactness. This algorithm constructs a whitening matrix using the FTN-ISI matrix to suppress noise correlation, and then performs detection through iterative linear and nonlinear estimation. Simulation results validate that the WE-OAMP algorithm outperforms benchmark algorithms in terms of bit error rate performance, especially in coded systems. Furthermore, we derive the achievable capacity of FTN signaling with WE-OAMP detection, demonstrating capacity improvement compared to Nyquist systems.