Quantum–Classical Convolutional Neural Network for Robust Orbital Angular Momentum Demodulation in Turbulent Channels

ABSTRACT We present a hybrid quantum–classical convolutional neural network (QC‐CNN) for classifying orbital angular momentum (OAM) modes in free‐space optical (FSO) channels affected by atmospheric turbulence. The approach embeds a convolutional quantum (quanvolutional) preprocessing layer that encodes 2 × 2 image patches into parameterized quantum circuits with rotation and entangling operations, yielding quantum‐extracted features that feed into conventional convolutional layers. We employ a two‐stage training protocol: large‐scale pretraining on 940 000 intensity images spanning a range of turbulence strengths, followed by task‐specific fine‐tuning with selective layer freezing. Turbulence was modeled using a modified Hill–Andrews phase screen and evaluated at multiple strengths over 1000 m propagation distance. The hybrid model achieves high classification accuracies (>97% across strong turbulence scenarios) and reduces bit‐error rates (BER) from up to 0.11 without pretraining to 0.001–0.08 after quantum‐layer pretraining. Results demonstrate that quantum‐inspired preprocessing combined with transfer learning significantly improves resilience to severe atmospheric distortions and offers a scalable route toward real‐time, high‐fidelity OAM demodulation in FSO systems.