Improving the Measurement of Knowledge in Multiple-Choice Tests: A Hierarchical Multinomial Processing Tree Approach

In multiple-choice (MC) tests, correct item responses can stem from actual mastery as well as from lucky guessing. Several approaches have been developed to address this problem, including the implementation of a “Don’t know” option. However, scoring such MC tests is challenging, as traditional scoring techniques are differentially biased against certain response strategies. In this research, we propose a hierarchical multinomial processing tree (MPT) model that allows measuring the latent ability while taking into account the tendency to engage in guessing and the preference for a specific response option when guessing. The proposed MPT model includes crossed-random person and item effects and can be estimated in a Bayesian framework. We applied the model to an empirical dataset of a general knowledge test with a true/false response format and “Don’t know” option. The model showed good fit to the data and revealed considerable variability in model parameters across both persons and items. Moreover, we found evidence for an improved measurement of knowledge in the form of higher convergent validities compared to conventional scoring techniques. These results highlight the usefulness of applying hierarchical MPT modeling to psychometric questions. This OSF project contains the JAGS model syntaxes, the materials of the empirical application (dataset, R script for model estimation), as well as the simulation code.