DMRN+14: Digital Music Research Network Workshop Proceedings 2019

While recent developments in MIR have proven automatic analysis of content-based similarity to be reliable, it remains unclear whether these algorithms can be meaningfully applied to cross-cultural analyses of more diverse samples.There is thus a need to evaluate automatic methods with perceptual ground truth data.Here we conducted multiple perceptual ratings tests on a subset of the Cantometrics recordings across 62 participants.We compared evaluating perceptual similarity by pairwise comparison, an odd-one-out method and a simplified feature evaluation based on 6 Cantometric features (namely, Ornamentation, Vocal Range, Tempo, Rhythmic Regularity, Vocal Tension, and Vocal Texture).Preliminary analysis shows that while the same person may disagree with themselves depending on the evaluation method, groups in general tend to agree with each other for the same method.We then compare the perceptual similarity ratings against existing automated algorithms and find minimal correlation, suggesting that there is still much room for improvement in automated cross-cultural content-based similarity.