Drug approvals in crowded drug classes potentially reflect efficacy overestimates and false positivity: A portfolio analysis of immune checkpoint inhibitor trials

OBJECTIVES: Institutions making decisions regarding drug development, approval, or insurance reimbursement require accurate estimates of drug efficacy that reflect the drug's true value. Stein estimation and multiplicity correction can be applied to trials for an entire class of drugs, minimizing random error in effect size estimates. STUDY DESIGN AND SETTING: We downloaded ClinicalTrials.gov registry entries for trials testing immune checkpoint inhibitor (ICI) drugs, including only trials with overall survival (OS) or progression-free survival (PFS) as a primary outcome, in phases 1-3, with 100 patients or more, at least one US site and primary completion date before January 1, 2023. Effect sizes and P values were recalculated for pivotal trials. A change was considered "meaningful" if the adjusted hazard ratio (HR) crossed the null or an European Society for Medical Oncology (ESMO)-Magnitude of Clinical Benefit Score (MCBS) cutoff. RESULTS: We included 14 ICIs, eight of which were approved by the Food and Drug Administration (FDA). These were tested in 111 trials, 45 of which led to FDA approvals. Among FDA-approved drugs, 82 distinct OS/PFS HRs were reported as primary outcomes supporting an FDA label approval/revision. After recalculation, estimates improved to a meaningful degree for one pivotal trial (1%) but diminished meaningfully for 29 (35%). Among OS/PFS HRs, 74 (90%) had an original P value that was less than 0.05. Among these, 20 (27%) had a corrected P value that lost significance. CONCLUSION: When statistically adjusting for the totality of trials testing ICIs, effect sizes in pivotal trials generally decreased, and some P values lost significance. This may have implications for minimizing error in approval, reimbursement, and certain clinical decisions. PLAIN LANGUAGE SUMMARY: Drug approval and insurance reimbursement require accurate estimates of drug efficacy. In cases where drugs are tested in many different trials, statistical methods can be used to minimize error among efficacy estimates. In this paper, we use a statistical method to minimize error in calculations of the efficacy of drugs in a class of cancer drugs known as "immune checkpoint inhibitors" (ICIs). After this adjustment, there was a meaningful improvement in the estimate of efficacy for one (1%) of the clinical trials of ICIs that were used to justify FDA approvals, but estimates of efficacy for 29 (35%) of pivotal trials meaningfully worsened. We also corrected for potential false positives among trials in our sample. Among the 74 (90%) drug trials that had a significant result (P < .05), 20 (27%) lost significance after correction. The analyses performed herein have potential for reducing the potential bias in efficacy estimates used for making policy.