Spectrally Efficient Time-Frequency Space Modulation: A Delay-Doppler Domain FTN Waveform for ISAC Systems

This paper presents the spectrally efficient time–frequency space (SETFS) modulation, a two-dimensional (2D) format extending spectrally efficient frequency division multiplexing (SEFDM) and orthogonal time–frequency space (OTFS) by introducing a controlled spectral overlap across both subcarriers and sub-bands to enhance spectral efficiency. Periodicity in time and frequency is preserved through a double cyclic prefix, enabling simple equalization while avoiding 2D inter-symbol interference. However, the presence of a spectral overlap induces 2D inter-carrier interference, that makes symbol detection more complicated. To mitigate channel estimation errors and also support high-resolution sensing, a novel channel estimator, dubbed Newton-based estimation and spectral cancellation algorithm (NESCA) is proposed. Numerical results show that SETFS improves the achievable rate by up to 19% over OTFS, while NESCA achieves near-optimal delay–Doppler estimation, offering an effective balance between spectral efficiency, receiver complexity, and sensing accuracy.