On Representational Alignment Across Tokenization Schemes in Pretrained Transformers

Subword tokenization is a foundational component of modern language models, yet its relationship to learned representation geometry is rarely examined directly. We present an empirical comparison of pretrained transformer encoders that employ different tokenization strategies, including WordPiece, byte-pair encoding (BPE), and SentencePiece. Using frozen embeddings, linear probing, cross-space linear alignment, and Centered Kernel Alignment (CKA), we quantify downstream separability, transfer degradation, and geometric similarity across models. We observe consistent performance drops when transferring between tokenizer spaces and moderate similarity scores under CKA, suggesting that representations learned under different segmentation schemes are not fully interchangeable. We frame this work as a measurement study and baseline for future controlled investigations into tokenizer-induced representational effects.