RDoFlow: Automatically assessing under-specified statistical analyses in HCI

When designing and analyzing a study, researchers must navigate a large space of methodological decisions, or “researcher degrees of freedom.” If these choices are not preregistered or transparently reported, they can increase the risk of inflated false-positive rates and exaggerated effect sizes, undermining scientific credibility. Drawing on psychology research that characterizes these degrees of freedom, we create a protocol for scoring how hypotheses are reported in the HCI literature (ReportDoF). We manually apply ReportDoF to 100 hypotheses from HCI texts authored between 2015-2025, including both preregistrations and papers. Based on this experience, we contribute an LLM workflow and proof-of-concept interactive interface (RDoFlow) that applies ReportDoF to new texts, enabling large-scale analysis of the composition and quality of reported analysis specifications. For example, RDoFlow reveals that HCI research more frequently tests multiple dependent variables for a single hypothesis than psychology research does—a practice that increases the risk of false positives.