Autoencoder-based nonlinear impairment mitigation in high-capacity four-channel wavelength division multiplexed long-haul optical communication systems

The exponential growth of Internet traffic necessitates the use of advanced modulation formats and dense wavelength division multiplexing (WDM) to maximize the spectral efficiency of optical fiber communication systems. However, the accumulation of chromatic dispersion and fiber nonlinearities, specifically self-phase modulation and cross-phase modulation, severely limits the performance of high-order quadrature amplitude modulation (QAM) formats in long-haul links. We propose a physically grounded end-to-end autoencoder (AE) framework for dual-polarization (DP) WDM systems that jointly optimizes a geometric constellation shaping encoder and a deep neural receiver. We implement a scalable four-channel WDM system operating at 32 Gbaud per channel over a 1000-km link modeled by the Manakov split-step Fourier method. We systematically compare three receiver architectures: a linear electronic dispersion compensation baseline, a per-channel two-dimensional convolutional neural network (2D-CNN) AE decoder, and a joint 4-WDM 2D-CNN AE decoder. Numerical results demonstrate that although the per-channel AE effectively mitigates intra-channel nonlinearities for 16-, 32-, and 64-QAM, the joint architecture suffers from scalability issues at higher modulation orders. The proposed per-channel 2D-CNN receiver achieves a bit error rate (BER) below 2×10−2 for DP-64-QAM, significantly outperforming the linear baseline, which fails with a BER of ≈0.5 due to unmitigated fiber nonlinearities.