A Comparative Study of SMILES, SELFIES, and ECFP4 Representations for Molecular Similarity Search

We compare SMILES and SELFIES embeddings generated by two transformer models (ChemBERTa, SELFormer) against 1024-bit ECFP4 fingerprints for molecular similarity search. Using thirty chemically diverse query molecules, we retrieve the top-k nearest neighbours k ∈ {5, 8, 10, 15, 20, 25, 35} via cosine similarity for all embedding representations and Tanimoto's coefficient for fingerprints. Hits are evaluated on chemical similarity (Hamming distance), structural similarity (Graph Edit Distance, GED), and functional similarity (Functional Hamming Distance). Statistical significance is assessed with one-way ANOVA and Tukey's HSD. Our results confirm that ECFP4 with Tanimoto yields the lowest mean Hamming distances (mean=12.5 at k=35) as well as the lowest mean GEDs (mean=18.9). SELFIES embeddings consistently outperform SMILES embeddings yet remain inferior to fingerprints.