What Pilot Studies Can (and Cannot) Do for Validity in Psychological Research

Recent debates about psychology’s credibility have renewed attention to validity threats. We argue that pilot studies can function as methodological due diligence by improving early decisions about design, measurement, manipulation, sampling, and analysis prior to a main study. We examine four types of validity: construct, internal, external, and statistical-conclusion validity. Across these domains, we show how piloting can help researchers assess whether materials and procedures function as intended, identify potential validity threats, and provide key design parameters for planning and model choice. However, we also caution that piloting can harm validity when treated as a consequence-free space in which design or analytic choices are made without disclosing the pilot evidence behind them, or when small, noisy pilots are used to justify overconfident inferences. We therefore argue that when pilots are used to justify substantive design or analytic choices, they should be treated as part of the evidential record and reported transparently. Greater transparency would enable readers to evaluate how pilot evidence shaped the final design of a study and, consequently, better assess the validity of its findings.