Nonlinear compensation using a robust neural network for noisy samples in a high-speed coherent optical transceiver

Nonlinear impairments in coherent optical transceivers are the primary bottleneck for enhancing transmission capacity. However, as most of the recently prevalent nonlinear equalizers (NLE) are data-driven, their dependence on training signal quality and the incompatibility between their machine-learning frameworks and communication scenarios have contributed to the potential risks of their applications. Therefore, this paper aims to improve the deficiency of NLEs and enhance their practical possibilities. We focus on the neural network (NN)-based nonlinear equalizers (NNLE) and extend the meta-training methods to improve the training methodology in the basic NLEs' framework. A noisy-samples-robust NN (NSNN) based NLE architecture is proposed. With a bilevel optimization based on meta-learning to train an affiliate model for noisy samples purification, the proposed NSNN can be efficiently trained under data sets with a sufficient amount of noisy samples mixed in. Thus, it can provide reliable performance in many scenarios where traditional NLEs' performance is limited. We experimentally demonstrate its effectiveness in nonlinear impairments in a high-speed coherent optical transceiver, and its performance is compared with representative traditional NLEs. In a dual-polarization probabilistic constellation shaping 128 QAM back-to-back transmission with a net bitrate over 1 Tbps, the proposed NSNN can provide an optical signal-to-noise (OSNR) gain of over 1.5 dB at the 24% soft-decision forward error correction threshold of 4.5e-2. The results also illustrate that the proposed NSNN can provide stable gains at acceptable complexity cost under various test conditions, and achieve more stable generalizability than common NLEs.