miLLi: Model Integrating Local Linguistic Insights for Morphologically Robust Tokenization

Standard statistical tokenization algorithms often struggle to preserve the morphological boundaries of agglutinative languages such as Azerbaijani. This study introduces miLLi 1.0, a hybrid tokenizer that integrates a rule-based root dictionary with a statistical Byte-Pair Encoding (BPE) approach. The model’s distinguishing feature is a dynamic phonological restoration algorithm designed to map allomorphic variations back to their canonical root forms. Empirical evaluations on the Tatoeba corpus demonstrate that miLLi 1.0 (with a 32,000 vocabulary size achieving 1.980 T/W) outperforms global standards such as GPT-4o and mBERT in terms of representation efficiency. While exhibiting lower token density compared to local statistical models, miLLi 1.0 demonstrates high linguistic robustness, achieving 58.0% in Morphological Boundary Accuracy (MBA) and 79.0% in Root Consistency Rate (RCR), significantly outperforming a standard BPE baseline trained under identical conditions (47.0% MBA and 1.0% RCR). The findings suggest that the integration of a linguistic filtration layer establishes an optimal balance between statistical compression and semantic integrity.