Tokenizing 3D Molecule Structure with Quantized Spherical Coordinates

While language models (LMs) have demonstrated remarkable general-purpose capabilities across domains, including molecule generation using line notations such as SMILES and SELFIES, their direct application to 3D structure design remains constrained by two interdependent challenges. First, the difficulty in designing a 3D line notation that ensures SE(3)-invariant atomic coordinates and supports autoregressive generation. Second, the incompatibility between continuous spatial coordinates and the discrete token inputs required by LMs. To address this, we propose Mol-StrucTok, a unified framework for tokenizing 3D molecular structures. Our approach comprises two key innovations: (1) a 3D line notation—Spherical Coordinate Notation—that encodes local atomic environments in spherical coordinates, agnostic to 2D notations and inherently SE(3)-invariant; and (2) a structure-aware Vector Quantized Variational Autoencoder (VQ-VAE) for discretizing these coordinates into chemically valid tokens suitable for language model processing. Leveraging this tokenization framework, we train a GPT-2 style model for end-to-end 3D molecular generation. Empirical results demonstrate strong, task-dependent performance: in unconditional generation, Mol-StrucTok achieves diffusion-level stability with ~28× faster inference; in conditional generation, it reduces property-matching mean absolute error (MAE) by 5–8× compared to diffusion-based methods, highlighting the advantage of autoregressive contextual modeling for precise control of molecular attributes. Our code is available at https://github.com/KyGao/Mol-StrucTok.