Clinical validation of large-scale functional assays: CanVIG-UK recommendations for truthset assembly based on 2122 gene-truthset-assay evaluations

Background: Clear guidance is lacking regarding selection of ‘truthsets’ variants to be used in clinical validation of functional assays, in particular ensuring reliable assay predictiveness for missense variants. Surveys indicate that inconsistency in clinical validation of assays reduces confidence amongst diagnostic scientists in leveraging large-scale functional assays to advance clinical variant classification. Methods: We firstly reviewed 112 sets of ClinGen gene-specific classification specifications (CSPECs) to assess variability in methodology for truthset assembly and clinical validation of assays We then delineated how truthsets of ClinVar classifications might variously be assembled, applying differing rules regarding variant type and stringency of classification. We further considered options for truthset augmentation with additional benign missense variants systematically-generated using ACMG/AMP rules (of differing stringencies). In total, these constituted 70 basic approaches to ClinVar-based truthset construction. We applied these for each of VHL, BRCA1, BRCA2 and RAD51C, censoring ClinVar for the date of assay publication. We additionally analysed how differing phenotypes for ClinVar classification submissions impacted truthset size. We then performed clinical validation of five large-scale multiplexed functional assays for VHL, BRCA1, BRCA2, and RAD51C using these truthsets. We also performed clinical validations against secondary thresholds of deleteriousness-neutrality, which were published for two of the assays. Results The ACMG-evidence-points (ACMG-EPs) for clinical validation of each assay varied widely according to which truthset was used. For example, examining 700 ClinVar-based truthsets against VHL assay outputs for 2,268 variants, ACMG-EPs for pathogenicity ranged from 0.62 (no evidence) to 7.31 (strong evidence) and for benignity from -1.50 (supporting evidence) -4.29 EPs (strong evidence). Truthsets comprising just missense variants typically resulted in lower EPs than nonsense-synonymous variants, more due to paucity of variants than lack of concordance. Truthsets based on more stringent ClinVar classifications or tighter phenotypic alignment typically improved concordance but lacked variant numbers. Conclusions Limited numbers of ClinVar missense classifications, especially of benignity, has been perceived as limiting. Assay validation should apply truthsets of missense-only variants, but power can be mitigated by augmentation with systematically-generated benign missense variants and selection of whichever missense truthset optimises ACMG-EPs. Pathogenicity and benignity validation should be considered separately.