Conformer-Based Nonlinearity Compensation for Coherent Optical Fiber Transmission Systems

Fiber nonlinear distortion remains a major limitation in high-speed coherent optical transmission systems, constraining both reach and capacity. Approaches to mitigate these effects include digital signal processing and machine learning. While recurrent neural networks are effective, their sequential nature limits parallelization, making them less suitable for high-speed hardware. Transformer-based nonlinear compensation (TRN-NLC) models offer a promising alternative by processing symbols in parallel and capturing long-range dependencies. Building on the TRN model, the Conformer (CNF) architecture augments the encoder with convolutional modules to better capture local features while modeling global dependencies. In this work, we propose a CNF-NLC model for dual-polarization (DP) coherent optical systems and benchmark its performance against a TRN-NLC model. We conduct extensive simulations on a DP 16-ary quadrature amplitude modulation system operating at 32 Gbaud. The signal is transmitted over 3200 km with launch powers ranging from −3 to +3 dBm. . The results demonstrate that the proposed CNF-NLC model significantly outperforms both conventional chromatic dispersion compensation (CDC) and the TRN-NLC model. The CNF-NLC model consistently delivers better performance, particularly at high launch powers, achieving up to 0.5 dBQ-factor improvement over the TRN-NLC model and up to 1.1 dB over CDC. The application of sparse attention masking introduces minimal performance degradation while enabling substantial resource savings. Additionally, we present a complexity analysis for both TRN-NLC and CNF-NLC models in terms of real multiplications per symbol, addressing the trade-off between performance and computational cost. Although the CNF-NLC model introduces modest computational overhead, the analysis shows that at comparable complexity the CNF-NLC model provides about 0.3 dB enhanced performance over the TRN-NLC model. This positions the CNF-NLC model as a scalable and parallelizable solution for real-time coherent optical receivers, offering a favorable balance between efficiency and accuracy.