Comparative Study of Transformer-Based Character-Level and Word-Level Text Prediction Models

Traditional character-level language models exhibit inherent limitations in text prediction tasks, primarily manifested as insufficient coherence in generated texts. This issue largely stems from the diversity of character and phrase combinations, where the same character or phrase may have different collocations across various contexts. To address these challenges, this paper proposes a phrase-level text prediction model based on the Transformer architecture. Unlike conventional character-level models, this approach treats phrases as the minimum input units by constructing N-gram features, thereby avoiding erroneous segmentations. The model generates subsequent text based on preceding word sequences composed of segmented phrases, producing one new phrase at a time during text generation to enhance overall fluency. Comparative experiments conducted on Chinese text datasets demonstrate that, compared to character-level models, the phrase-level model achieves reduced perplexity (PPL) and significantly improved BLEU scores. The generated texts better conform to natural language conventions and exhibit superior semantic coherence, validating the model’s superiority in text prediction tasks.