Spam Message Classification Using Transformer Architecture with Multi-Head Attention Mechanism

This paper investigates the application of a Transformer-based deep learning model with a multi-head self-attention mechanism for spam message classification. The proposed architecture adopts an encoder-only Transformer design that combines token and positional embeddings, stacked Transformer encoder blocks, and a global average pooling-based classification head. The model is trained and evaluated using the SMS Spam Collection dataset, which reflects real-world class imbalance between spam and legitimate messages. Experimental results demonstrate that the proposed model effectively captures contextual and semantic relationships within short text messages, achieving an accuracy of 97.8%, a precision of 96.5%, and a recall of 95.2%. Further analysis of attention weights indicates that the model focuses on key spam-indicative terms such as promotional keywords and numerical patterns, enhancing interpretability without explicit feature engineering. These findings confirm the effectiveness of Transformer architectures with multi-head attention for robust and accurate spam message classification.