Interobserver Variability Across Whole-Slide Imaging Systems

Context.—: Digital pathology whole-slide images (WSIs) must be of adequate quality for use in diagnostics and the development of artificial intelligence algorithms. Inadequate slides should be re-scanned. There is insufficient data on what qualifies as WSI adequacy. Objective.—: To report on the degree of inter-cohort agreement when evaluating WSI quality and adequacy for use in primary diagnostics. Design.—: The 60 validation slides were scanned by 3 scanners (Leica GT450, Hamamatsu NanoZoomer S360, and Huron TissueScope iQ). The resulting 180 WSIs were independently evaluated on a quality pass or fail basis by 12 observers representing 4 staff members of the following: in-house attending physicians, fellows, and imaging technicians. The Cohen's kappa was calculated overall and within each cohort, and the acceptance rate was compared between cohorts. Results.—: For almost every cohort on each scanner, the Cohen's kappa was weak (0.21-0.4) or very poor (