The Replication Crisis and the New Statistics

This chapter examines the replication crisis that emerged in psychology around 2016 and its implications for corpus linguistics research. The crisis arose when supposedly significant findings in prestigious journals failed to replicate, challenging fundamental assumptions about statistical significance and research methodology. The author examines a specific corpus linguistics controversy between Millar (2009) and Leech (2011) regarding modal auxiliary verb trends, demonstrating how conflicting ‘significant’ results can emerge from ostensibly similar studies. The chapter identifies six alternative hypotheses for replication failures: population differences, inappropriate pooling of meaning classes, selective representation of trends, unreliable per-million-word measures, violation of Binomial model assumptions, and speaker/text effects. The author notes how seemingly rigorous statistical methods can fail when underlying assumptions about data are violated, including the common practice of using word-based frequency baselines that do not conform to Binomial model requirements. Drawing on Gelman and Loken’s critique of ‘forking paths’ in data analysis, the chapter addresses the fundamental problem of post hoc hypothesis selection and multiple testing. The chapter explores how researchers may inadvertently engage in selective reporting, unconsciously ‘ p -hacking’, leading to overestimation of claim robustness. The chapter concludes with some practical recommendations including implementing replication steps, focusing on effect sizes rather than significance alone, systematic consideration of alternative hypotheses, and enhanced transparency in reporting.