Does the Use of Crowdsourced Listeners Yield Different Speech Intelligibility Results Than In-Person Listeners for Typically Developing Children?

PURPOSE: We examined the performance of crowdsourced listeners compared with in-person listeners on the measurement of speech intelligibility for typically developing children. We used three different in-task quality check criteria to screen listeners and examined between-listener intelligibility differences and interrater reliability under each criterion. We also examined how crowdsourced intelligibility results compared with in-person results. METHOD: = 300 total) and compared scores with in-person results from two listeners per child. We used intraclass correlation coefficients (ICCs) and computed pairwise differences among listener groups for each of three in-task quality criteria groups and the in-person group. We modeled intelligibility as a function of listener source (in-person vs. crowdsourced) and child age using mixed-effects regression with smoothing splines. RESULTS: Lower ICCs and larger between-listener differences were observed for crowdsourced compared to in-person listeners, regardless of in-task quality check criteria, but in-task quality check criteria reduced the disparity. Crowdsourced listeners produced intelligibility scores that were up to 7 percentage points lower than in-person listeners, even under the most stringent in-task quality check criterion. Results showed the same patterns of change with children's age as in-person listener findings. Children with midrange (65%-83%) intelligibilities were the most negatively impacted by the use of crowdsourced listeners. CONCLUSIONS: Rigorous in-task quality check criteria improved crowdsourced listener data. Speakers with midrange intelligibility were the most negatively impacted by the use of crowdsourced listeners, with an intelligibility difference of about 7 percentage points. Intelligibility data obtained with crowdsourced listeners should be interpreted with caution, and future studies should evaluate how crowdsourced intelligibility data differs from in-person data for disordered populations of speakers. SUPPLEMENTAL MATERIAL: https://doi.org/10.23641/asha.31101451.