MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Transactions of the Association for Computational Linguistics 2026 Open access

Read full text
Authors: Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza
DOI: 10.1162/tacl.a.600
Cited by: 1
References: 43

Abstract

Abstract We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.1

Fields of Study & Subjects

Natural Language Processing Techniques Language and cultural evolution Topic Modeling

Publication details

Journal / source
Transactions of the Association for Computational Linguistics
Publisher
MIT Press
Volume
14
Pages
193–216
Publication date
16 January 2026
Language
EN