BEAST: BYTE-WISE EMBEDDING ARCHITECTURE FOR SEMANTIC TRANSFER

In this work, we introduce BEAST (Byte-wise Embedding Architecture for Semantic Transfer), a lightweight, tokenizer-free static embedding model that operates directly on byte-level input. BEAST learns 256-dimensional word representations by distilling semantic knowledge from GPT-3, without relying on tokenization, subword vocabularies, or linguistic preprocessing. Trained on over 1.2 million Spanish words, BEAST outperforms classic static embeddings such as FastText, Word2Vec, and GloVe, as well as several multilingual contextual models, on the Spanish translation of the SimLex-999 benchmark. Despite its compact size (51MB), BEAST captures rich lexical semantics and generalizes well to out-of-vocabulary words. We also demonstrate its practical utility in semantic clustering tasks, showing its potential as a robust embedding backbone for low-resource and multilingual NLP pipelines.