Conceptualizing Surprise with the Jensen–Shannon distance: A Bayesian Information Theoretical Approach for the Social Sciences

Generally in the social sciences, a result is informally deemed ‘surprising’ if its associated p-value is sufficiently small. This implicit interpretation, however, is both conceptually and mathematically inappropriate, and such misuse of the p-value can lead to erroneous conclusions about the novelty of results. To solve that issue, this paper builds on Bayesian inference, Information theory, and considerations specific to the social sciences, to argue for the adoption of a more appropriate conceptualization of surprise as the relative entropy between prior and posterior knowledge. Novel to the social sciences, this formal conceptualization enables researchers to appropriately measure surprise as the Jensen-Shannon distance, for which the paper contributes with easily implementable software and a demonstration of its use in relation to empirical data.© Keywords: Surprise; Novelty; Jensen-Shannon distance; JS distance; Jensen-Shannon divergence; JS divergence; Relative entropy; Kullback-Leibler divergence; KL divergence; Distance; Divergence; Dissimilarity; Differential entropy; Entropy; p-value; S-value; Information theory; Bayesian inference