MolGAN-QRL: a hybrid framework for molecule generation using quantum-enhanced reinforcement learning

Discovering novel drug candidates remains a considerable challenge in pharmaceutical research. Generative AI models such as Generative Adversarial Networks (GANs) have shown considerable promise in de novo molecular generation. They demonstrated high potential in drug discovery applications, yet they often face challenges such as limited chemical coverage and mode collapse. In the present study, we developed MolGAN-QRL, a hybrid quantum-classical framework that introduced quantum-enhanced reinforcement learning within the MolGAN architecture to address these limitations. The proposed framework leveraged a hybrid reward mechanism to further optimize chemical validity, uniqueness, and drug-likeliness of the generated molecules. Experimental results demonstrated that MolGAN-QRL consistently achieved enhanced generative performances compared to classical MolGAN, with up to a 16-fold increase in the count of unique and valid generated compounds under certain conditions. These gains reflected the effectiveness of quantum-guided exploration and highlighted the known trade-off between uniqueness and validity in generative chemistry. Overall, our findings underlined the value of quantum-enhanced reward modeling in mitigating mode collapse and advancing molecular generation, and support the potential of hybrid quantum-classical methods to advance generative chemistry for drug discovery applications. SCIENTIFIC CONTRIBUTION: MolGAN-QRL introduces the first variant of the MolGAN framework, that is augmented with a variational quantum circuit (VQC) within the reinforcement-learning reward module, rather than the generator, the discriminator or the noise function. It leverages a hybrid reward mechanism that trains a quantum-classical function that lead to better mitigation of mode-collapse, through higher uniqueness scores and 16-fold more novel, valid and unique molecules generated.