A Weighted Self-Attention Optimization Framework for Transformer-Based Text Classification

Transformer models have significantly advanced Natural Language Processing (NLP) by replacing recurrence with self-attention mechanisms [1].While standard Transformers compute attention uniformly across heads and tokens, this may dilute task-specific importance in classification problems [2,8].This paper proposes a Weighted Self-Attention Optimization (WSAO) framework that introduces adaptive token-level weighting into the Transformer encoder to enhance discriminative feature learning.Using a public benchmark dataset, we demonstrate that the proposed formulation improves classification performance compared to baseline Transformer [1] and LSTM models [3].Mathematical formulation, experimental evaluation, and comparative analysis are presented to highlight the effectiveness of the approach.