Bounds on kappa-error plots for classification diversity analysis

Classifier ensembles benefit in terms of both accuracy and size reduction from the diversity of their members. It has become popular to analyze diversity using so-called kappa-error plots. In 2013, Kuncheva presented a bound on kappa-error points that identifies regions of the diagram that can never be occupied. She also claimed that there is an unoccupied space in the vicinity of these bounds where new ensembles might be engineered. This article presents a more in-depth study of the areas that may and may not be occupied by kappa-error points. It is shown that the region that Kuncheva and some followers claimed to be available may in fact be unavailable in many practical applications. Several properties are proved that constructively characterize the contents of kappa-error optimal contingency tables describing class assignment differences between raters in multiclass problems with arbitrary class proportions. Formulae for the exact bounds of the plots are provided. Examples with equal class sizes are also analyzed to illustrate and visualize how the availability of these areas depends on the number of classes. The analysis reveals and discusses further weaknesses of the kappa statistic as a measure of model diversity. Recommendations are provided for researchers who wish to base model selection on kappa or to prepare, analyze, and interpret kappa-error plots based on the location of point clouds.