Spelling Bee Embeddings for Language Modeling
Abstract
We introduce a simple modification to the embedding layer. The key change is to infuse token embeddings with information about their spelling. Models trained with these embeddings improve not only on spelling, but also across standard benchmarks. We conduct scaling studies for models with 40M to 800M parameters, which suggest that the improvements are equivalent to needing about 8% less compute and data to achieve the same test loss.
Fields of Study & Subjects
Topic Modeling
Natural Language Processing Techniques
Speech Recognition and Synthesis
Publication details
- Journal / source
- arXiv (Cornell University)
- Publication date
- 25 January 2026
- Document type
- Preprint