Embedding-Based Applicability Domain for Graph Neural Networks: A Comprehensive Benchmark on Molecular Property Prediction

Graph Neural Networks (GNNs) are powerful tools for molecular property prediction, but they are not magic. When applied to molecules unlike their training data, they produce unreliable predictions that are difficult to detect. The Applicability Domain (AD) concept addresses this by defining regions of chemical space where predictions should be trusted. We benchmark embedding-based AD methods across three GNN architectures (GCN, GAT, MPNN) and nine MoleculeNet datasets. Our hybrid AD score combines density estimation (KDE), distance metrics (kNN, Mahalanobis), and prediction uncertainty (entropy, margin). An automated thresh- old optimization selects cutoffs that balance accuracy gains against sample retention. Benchmark results confirm the utility of the framework. Restricting predictions to the In-Domain subset improved AUC up to 0.96 (BACE-GCN) and reduced RMSE to 0.66 (Lipo-GAT), while Out- Domain samples showed 2-3× higher error rates. Coverage ranged from 8% (aggressive filtering) to 99% (conservative), depending on dataset difficulty. Case studies on ESOL and BBBP demonstrate that flagged molecules have interpretable structural features (long aliphatic chains, unusual ring systems, or heteroatom combinations) that correlate with prediction failures.