Towards a more informed and balanced use of scientific performance metrics

Abstract The goal of scientific assessment is to predict which individuals can make optimal use of limited resources within a specific context to make optimal allocation decisions. In academic contexts that pertain to individual-level allocations, this is most relevant for decisions on whom to hire for academic positions, nominate for awards, or whose research projects to fund. The current perspective paper draws upon insights from decades of psychometric research and more recent research on scientific performance to derive a set of five psychometric criteria that should be met for optimal assessment procedures in academia. Although data-driven decision making has gained popularity in most domains, there is increasing resistance against using quantitative measurements in scientific assessment. Recently, several stakeholders have proposed to jettison such measurements and focus instead on qualitative indicators or narratives. We argue that both quantitative and qualitative assessment do not always meet our five criteria, but solely relying on qualitative indicators appears to be a suboptimal strategy. We argue instead that there are smarter ways to use quantitative indicators so that they become more reliable, predictive, and ultimately also more efficient and equitable. We conclude with a set of recommendations for scientific quality assessment that is based on the most recent psychometric and scientific insights. In an appendix, we apply these recommendations to a Dutch case study of how researcher information is considered in the application procedure for a prestigious individual grant.