End-to-end of Tomlinson–Harashima precoding for faster-than-Nyquist coherent WDM transmission

We propose and experimentally demonstrate an end-to-end (E2E) Tomlinson-Harashima precoding (THP) scheme for faster-than-Nyquist (FTN) coherent WDM transmission. A complex-valued conditional generative adversarial network (CV-CGAN) trained on measured complex waveforms is used as a differentiable channel emulator, enabling joint optimization of the transmitter feedback equalizer (FBE) and receiver feed-forward equalizer (FFE). The proposed scheme is experimentally validated in a 30-channel FTN-WDM system over 80-km single-mode fiber (SMF). The experimental results demonstrate that the proposed CV-CGAN-based E2E THP scheme achieves the same bit error rate (BER) performance with optical signal-to-noise ratio (OSNR) sensitivity improvements of 0.5 dB and 1.5 dB compared to conditional generative adversarial network (CGAN)-based E2E THP and traditional minimum mean-square-error (MMSE)-based THP scheme, respectively, under a 20% soft-decision forward error correction (SD-FEC) threshold.