Leveraging Machine Learning Models to Predict Probability of Technical and Regulatory Success

Abstract Purpose Probability of technical and regulatory success (pTRS) is a key driver of risk-adjusted valuation and portfolio decisions, yet many assessment methods of pTRS depend on proprietary data or complex, hard-to-replicate architectures. We aimed to develop a reproducible, interpretable pTRS assessment method using only public trial features and open labels. Methods We extracted protocol-section features from ClinicalTrials.gov and paired them with binary outcome labels from the HINT clinical-trial-outcome repository (IQVIA-derived; small-molecule trials through 2021). Structured protocol variables were cleaned and one-hot encoded. Trials were mapped into 21 therapeutic areas using LLM-assisted classification of conditions/keywords. Data were split temporally: training included trials with primary completion dates before January 1, 2017 (n = 10,224) and testing included trials with start dates after January 1, 2017 (n = 478). To assess the impact of temporal separation on model performance, models were additionally trained and evaluated using datasets constructed with varying year gaps (0, 1, 3, and 5 years) between the training and test periods. We evaluated logistic regression, random forest, and XGBoost using ROC-AUC, PR-AUC, and Brier score, plus benchmark-relative comparison. Interpretability was assessed with SHAP. Results Random forest produced the strongest aggregate performance (ROC-AUC 0.788; PR-AUC 0.794; Brier 0.196), improving upon baseline model and the benchmark comparator across most settings. Sponsor type, phase, therapeutic area, and key design variables were consistently influential. Conclusion Interpretable models trained on open ClinicalTrials.gov protocol features can generate calibrated, decision-useful pTRS estimates that outperform standard benchmarks, offering a practical prior to complement expert elicitation in valuation and portfolio workflows.