Trainable, Multiword-aware Linguistic Tokenization Using Modern Neural Networks

Read full text Summarise with VIRAT
Authors: Clara Boesenberg, Kilian Evang
Cited by: 0
References: 27

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

We revisit MWE-aware linguistic tokenization as a character-level and token-level sequence labeling problem and present a systematic evaluation on English, German, Italian, and Dutch data.We compare a standard linguistic tokenizer trained without MWE-awareness as a baseline (UDPipe), a character-level SRN+CRF model (Elephant), and transformerbased MaChAmp models trained either directly on gold character labels or as token-level postprocessors on top of UDPipe.Our results show that the two-stage pipeline -UDPipe pretokenization followed by MaChAmp postprocessing -consistently yields the best accuracy.Our analysis of error patterns highlights how different architectures trade off over-and undersegmentation.These findings provide practical guidance for building MWE-aware tokenizers and suggest that postprocessing pipelines with transformers are a strong and general strategy for non-standard tokenization.

Fields of Study & Subjects

Natural Language Processing Techniques Topic Modeling Speech and dialogue systems

Publication details

Journal / source
Conference of the European Chapter of the Association for Computational Linguistics
Pages
266–276
Publication date
1 January 2026
Language
EN