Which Pieces Does Unigram Tokenization Really Need?
Abstract
The Unigram tokenization algorithm offers a probabilistic alternative to the greedy heuristics of Byte-Pair Encoding.Despite its theoretical elegance, its implementation in practice is complex, limiting its adoption to the SentencePiece package and adapters thereof.We bridge this gap between theory and practice by providing a clear guide to implementation and parameter choices.We also identify a simpler algorithm that accepts slightly higher training loss in exchange for improved compression.github.com/sanderland/script_tok
Fields of Study & Subjects
Algorithms and Data Compression
Artificial Intelligence in Games
semigroups and automata theory
Publication details
- Journal / source
- Findings of the Association for Computational Linguistics: ACL 2026
- Publisher
- Association for Computational Linguistics
- Pages
- 6351–6360
- Publication date
- 1 January 2026