Design of peptides with noncanonical amino acids using flow matching

MOTIVATION: The canonical vocabulary of 20 amino acids limits the chemical space available to proteins and peptides. Expanding this vocabulary to hundreds of noncanonical amino acids (ncAAs) allows the engineering of proteins with novel function and activity, and is of particular interest for therapeutic peptides such as macrocycles, where ncAAs can improve proteolytic stability, membrane permeability, and immunogenicity. However, existing structure-based design tools either cannot model ncAAs at all, or are restricted to a small, fixed vocabulary of ncAAs seen during training, and ncAA data in the Protein Data Bank is scarce and heavily biased. RESULTS: We present NCFlow, a flow matching generative model that places any arbitrary ncAA into a given protein backbone using only its atom types and bond connectivity, and therefore generalizes to ncAAs never seen during training. To supplement sparse training data in the Protein Data Bank, NCFlow is pretrained on millions of small molecule structures and a large set of protein-ligand complexes before finetuning on native noncanonicals found within proteins. NCFlow outperforms AlphaFold3-based methods in the structure prediction of unseen ncAAs. We further present a peptide design pipeline akin to in silico deep mutational scanning, and propose a scoring strategy combining deep learning-based and molecular dynamics-based alchemical binding free energy calculations to identify improved peptide variants. We apply the method on four protein-peptide complex test cases, and observe that incorporating noncanonicals can improve predicted binding affinity by up to -7.0 kcal/mol. AVAILABILITY AND IMPLEMENTATION: NCFlow is freely available at https://github.com/mjslee0921/ncflow.