Towards reliable data augmentation in machine learning: Practices to prevent data leakage

Data leakage, the unintentional use of illegitimate information during model training, fundamentally undermines the validity of machine learning evaluations. The improper sequencing of data augmentation relative to dataset splitting is a critical and pervasive manifestation of this problem that is frequently overlooked in practice. This pre-split application of augmentation violates the core independence assumption between training and test data, allowing models to memorize augmentation artifacts rather than learn generalizable features. Through rigorous reproducibility case studies in agricultural computer vision, we quantify the severe inflationary effect of this methodological flaw. Re-evaluating a recent study on tomato disease detection with a correct split-then-augment protocol reveals a performance drop from 98.70% to 32.77% mAP50, a reduction of 65.93 percentage points. A second case study on rice pest detection confirms this pattern, with a performance decrease from 99.5% to 89.1% after correcting for data leakage and near-duplicates. This analysis identifies two distinct leakage pathways, categorized as process-induced (improper augmentation sequence) and data-inherent (undetected duplicates in benchmarks). We demonstrate that this methodological error is perpetuated by educational gaps, tool design, and academic incentives. To mitigate these risks, we propose a comprehensive set of actionable best practices for researchers, reviewers, and educators. These include rigorous data integrity protocols and enhanced methodological checklists in peer reviews, as well as a paradigm shift towards pitfall-based learning. Our work highlights that preventing data leakage is not merely a technicality but a prerequisite for reliable and reproducible machine learning, with implications for computer vision and beyond.