Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors

Large language models (LLMs) are increasingly deployed in multilingual, real-world applications with user inputs -naturally introducing typographical errors (typos).Yet most benchmarks assume clean input, leaving the robustness of LLMs to typos across languages largely underexplored.To address this gap, we introduce MULTYPO, a multilingual typo generation algorithm that simulates human-like errors based on language-specific keyboard layouts and typing behavior.We evaluate 18 opensource LLMs across three model families and five downstream tasks spanning language inference, multi-choice question answering, mathematical reasoning, and machine translation tasks.Our results show that typos consistently degrade performance, particularly in generative tasks and those requiring reasoning -while the natural language inference task is comparatively more robust.Instruction tuning improves clean-input performance but may increase brittleness under noise.We also observe language-dependent robustness: high-resource languages are generally more robust than lowresource ones, and translation from English is more robust than translation into English.Our findings underscore the need for noise-aware training and multilingual robustness evaluation.We release a Python package for MULTYPO and make the source code publicly available at https://github.com/cisnlp/multypo. Error Example Sentence NoneColorless green ideas smell furiously.Replacement Colorless green ideaa smell furiously.Insertion Colorless greenm ideas smell furiously.Deletion Coorless green ideas smell furiously.Transposition Colorless green ideas smell furioulsy.