Threshold-Based Identification of non-Gaussian Distortion in Optical Constellations Using Clustering Validity Metrics

This work presents a threshold-based de-modulation strategy for 16-QAM and 4+12 PSK constellations impaired by non-Gaus-sian distortions. The proposed method uses clustering validity indices as a deci-sion metric. By applying fragmentation through clustering algorithms ‒k-means, fuzzy c-means (FCM), and Gustafson-Kes-sel FCM (GK-FCM)‒ we were able to iden-tify ellipsoidal distortions on external da-ta-symbol clusters and dynamically select appropriate demodulation strategies. The proposed clustering-based approach does not require IQ (in-phase and quadrature) branch imbalance and phase-offset correc-tions by redefining decision regions based on cluster centroids. We introduce the use of clustering validity indexes (Partition Coefficient, Separation, Xie and Beni’s, and Dunn Index) to characterize symbol distortion levels in constellation diagrams and establish performance thresholds. The combination of DI and XB provides a criterion for defining the threshold of non-Gaussian distortion. In particular, XB ≥ 10.7 and DI ≥ 0.015 may serve as em-pirical indicators that the constellation in radio over fiber (RoF) optical systems has transitioned into a more structured regi-me where the cluster centroids are used for demodulation. Experimental results show that at high-noise levels (the optical signal to noise ratio OSNR = 16 dB), the XBindex reaches its minimum value, confir-ming the method’s sensitivity to noise-in-duced distortion. Improvements in the optical signal-to-noise ratio (OSNR) of up to 2.1 dB for 16-QAM and 0.7 dB for 4+12 PSK were observed at a BER threshold of 10-² after transmission over 78.8 km of fi-ber. The combination of DI and XB indices provides a robust criterion for defining the threshold of non-Gaussian distortion. These experimental findings suggest that clustering validity metrics can serve as effective thresholds for adaptive demodu-lation, enabling real-time identification of non-Gaussian distortions in RoF commu-nication systems.Keywords: clustering, k-means, fuzzy c-means, Gustafson-Kessel, non-Gaussian distortion, nonlinear phase noise.