Mainstream Meta-Analysis of Clinical Trials Produces Strongly Inconsistent Estimators

Mainstream random-effects meta-analysis of clinical trials (IVW, weights inversely proportional to the estimated variance), as has been shown in two publications, cannot be trusted. This is especially disturbing as meta-analysis, the combination of like studies to produce an overall estimate of effect size, is at the apex of most evidence pyramids. The asymptotic distribution theory of the mainstream fails because weighted linear combination theory can only be applied to weights that are constants with high degrees of accuracy. In fact, they are certainly volatile random variables. The asymptotic setting requires that the number of studies being combined goes to infinity. Our novel finding is that the coverage of their “95% confidence interval” for overall effect size converges to zero (strong inconsistency). We have conducted reanalysis of about 30 highly influential meta-analysis of clinical trials and found that the mainstream methods had unsupportable qualitative conclusions in about 10%, leading to scientifically unsupportable public health policies. Further, about another 15% had unsupportable quantitative conclusions. The two references offer an asymptotically valid alternative, based on ratio estimation borrowed from survey sampling methods. We also show that if you adjust for the bias of the IVW estimate, you obtain the unpopular equally weighted estimate.