Three-pillar Evidence-based Methodological Framework for Experimental Machine Learning Studies

Reproducibility is essential for ensuring the trustworthiness and credibility of scientific findings. As a cornerstone of open science, it depends on rigorous methodology that enables not only the reproduction of results but also the validation of their appropriateness. In this paper, we propose, grounded in experience and theory, a concise, universal, and easy-to-implement methodological framework for experimental ML studies. The framework is built around three core components: datasets, quality measures, and experimental protocol, each of which is examined in detail. We introduce a practical tool for identifying methodological shortcomings: the Objective Assessment Rubric (OAR), designed to support consistent evaluation together with the guidelines on how to fill it. Furthermore, we conduct a retrospective analysis of several top-tier ML papers using our framework, revealing substantial methodological gaps. We believe the proposed framework can significantly enhance the credibility, transparency, and reproducibility of ML research (especially among early-career scientists), while remaining flexible enough to support future methodological advancements.