Scaffolding Responsible Software Use: Evaluating the Effectiveness of a Causal Inference Tool

Statistical methods depend on the satisfaction of assumptions about the data generating process in order for theoretical and demonstrated properties to hold. It is also implicitly assumed that methods are used as intended. This work presents evidence that, without strong scaffolding, the implicit assumption around proper implementation is frequently not satisfied and that substantial implementation gaps exist between methods’ theoretical properties and performance in applied contexts. We introduce thinkCausal, a novel causal inference tool, as an example of how scaffolded statistical software can increase the usability of statistical methods and reduce existing implementation gaps. We draw evidence from a randomized study where participants were asked to estimate a causal effect within one of three treatment arms: (1) using thinkCausal, a scaffolded point and click implementation of Bayesian Additive Regression Trees (BART) for causal inference, (2) using bartCause, an R package implementation of Bayesian Additive Regression Trees (BART) for causal inference, or (3) using a software and method of their own choosing. Results show that participants were only able to reliably obtain accurate results when using thinkCausal.