Post-hoc “omic” adjustment in clinical trials requires rigorous validation

To the Editor We commend Hull et al. for their elegant post hoc analysis of the ORBIT-3 and ORBIT-4 trials.1 Their work provides a compelling explanation for the discrepant outcomes of these replicate studies, powerfully demonstrating that unmeasured patient heterogeneity, particularly in the sputum microbiome and proteome, can act as a critical confounder in bronchiectasis clinical trials. The authors’ findings represent a significant step toward understanding the challenges of developing therapies for this complex disease. The central conclusion of the article—that adjusting for baseline differences reconciles the disparate treatment effects—hinges on the statistical method used for this adjustment. The authors note that an unsupervised dimension reduction technique (non-metric multidimensional scaling, NMDS), which captures overall variance in the microbiota, fails to align the trial results (rate ratio 0.81 vs 0.59). They subsequently employ a supervised method, linear discriminant analysis (LDA), which is specifically trained to find the linear combinations of taxa that best distinguish the 4 study arms. This approach successfully harmonizes the treatment effects to a rate ratio of ∼0.82. While this is a powerful exploratory strategy, the use of a supervised method for post hoc confounder definition warrants careful consideration. By defining the adjustment variables based on their ability to separate the outcome groups (in this case, trial arms), there is an inherent risk of overfitting and “double-dipping” the data.2 This practice involves using the same dataset first to generate a hypothesis (ie, discover the discriminating signature) and then again to test it, a circular process that can capitalize on chance associations present only in that specific sample. Such a ­procedure may identify and adjust for spurious, sample-specific variation rather than a true, generalizable biological confounder, potentially leading to a biased estimate of the treatment effect.3 This is a well-recognized challenge when integrating high-dimensional “omic” data into the analysis of randomized controlled trials.4 This point is not intended to diminish the authors’ important findings, but rather to highlight a crucial question for the field: how do we translate such a valuable post hoc discovery into a robust, pre-specified analysis for future pivotal trials? A critical step in this translation is rigorous validation to ensure the discovered microbial signature is not a statistical artifact. One approach is internal validation through resampling methods like k-fold cross-validation. Within the ORBIT dataset, a discriminating ­signature could be ­developed on a training subset, and its performance and impact on the treatment effect estimate could then be tested on a held-out validation subset.5 Repeating this process multiple times would provide a more honest assessment of the signature’s stability and true explanatory power, mitigating the risk of overfitting. Ultimately, however, the gold standard remains external validation.6 The LDA-derived signature from the ORBIT trials should be treated as a locked, pre-defined biomarker. Its utility would then need to be prospectively tested in a completely independent dataset; for example, from another clinical trial of inhaled ciprofloxacin or even a different anti-pseudomonal agent. If adjustment using this pre-specified signature clarifies the treatment effect in a new cohort, it would provide strong evidence that the signature represents a true biological endotype rather than a dataset-specific anomaly. In conclusion, Hull et al. have not only provided a plausible solution to the ORBIT trial paradox but have also illuminated a critical methodological crossroad for clinical research in respiratory medicine. Their analysis should catalyze an urgent discussion on establishing rigorous frameworks—from internal cross-validation to prospective external validation—for developing and implementing “omic-based” biomarkers for patient stratification and covariate adjustment in future trials. Supplementary material is available at the American Journal of Respiratory and Critical Care Medicine online. Please see the ICMJE disclosure forms, which have been provided as supplementary material.