GX-TGVAE: Transformer-based Graph Variational Autoencoder with GraphXForm for Chemically Valid Molecular Design

Deep generative models for molecular design frequently rely on string-based approaches using SMILES (Simplied Molecular Input Line Entry System). However, models with SMILES-based outputs (e.g., TGVAE) inherently suer from a trade-o between novelty and chemical validity of generated molecular data. Additionally, existing graph-based generative models (e.g., Graph-VAE) face computational challenges in capturing global dependencies. In contrast, GraphXForm provides an eective solution to these graph-based challenges. GraphXForm generates molecular graphs sequentially, enabling high chemical validity through mechanisms such as action masking. In this technical report, we propose GX-TGVAE (GraphXForm-based Transformer Graph VAE), a novel architecture that integrates the TGVAE encoder incorporating Transformer-based Graph Attention Networks with GraphXForm, a sequential graph construction decoder. This approach aims to mitigate the trade-o between novelty and chemical validity of SMILES-based generation inherent in TGVAE, and enables more diverse and novel molecular generation by connecting latent space to GraphXForm. Specically, the encoder utilizing a hybrid architecture of Transformers and GAT (Graph Attention Network) captures both global and local features of molecular graphs, compressing rich information into latent space. Based on the latent variable information, the Transformer-based decoder GraphXForm generates molecular graphs with powerful decoding capability possessing global understanding of the output, while enforcing valence constraints through action masking. This report describes the detailed methodology, mathematical formulation of loss functions, and future prospects. Source code is under development and will be publicly available at https://github.com/yutaPsaito/GX-TGVAE.