Quadratically Weighted Agreement Coefficients: Interpretations and Connections

In this study, we review interpretations and connections of chance-corrected agreement coefficients with quadratic weights, applicable when raters classify objects or subjects on an ordinal scale. Whereas correlation interpretations exist for coefficients with noninterchangeable raters, represented by Cohen's two-rater and Conger's multirater kappas, interpretations are essentially absent for interchangeable raters, represented by Fleiss' kappa and its two-rater version, Scott's pi. We show that Fleiss' quadratically weighted kappa equals Lin's generalized concordance correlation coefficient after recentering the rater means and covariance matrix at the grand mean. Furthermore, it equals the Pearson product-moment correlation and associated regression slope for two random raters, computed after concatenating the ratings from all possible rater pairs, including reversed rater order. Next, we demonstrate that Fleiss' and Conger's quadratically weighted kappas are linear transformations of each other, entirely determined by the pairwise differences in rater means and corresponding variances. As these kappas coincide if the rater means coincide, the conceptual distinction between interchangeable and noninterchangeable raters becomes empirically irrelevant if raters have (approximately) the same mean rating, even with substantially different rating distributions. Finally, Fleiss' quadratically weighted kappa (i.e., interchangeable raters) cannot exceed Conger's (i.e., noninterchangeable raters).