Transformer-Based Ensemble Model for Classification of Documents Based on English Vocabulary Words

Document classification remains a key challenge in natural language processing, particularly when dealing with complex vocabulary structures and imbalanced datasets. While traditional classifiers and individual deep learning models achieve moderate success, they often struggle to capture both local lexical patterns and long-range contextual dependencies. To overcome these limitations, we propose a Transformer-based Ensemble Model that integrates BERT with Convolutional Neural Networks (CNNs) and Bidirectional Long Short-Term Memory (BiLSTMs) networks. BERT embedding serves as the foundation, offering contextualized semantic representations. CNN extracts local n-gram features, while BiLSTM models sequential dependencies across the text. The outputs from these complementary models are combined using a stacking ensemble strategy with a meta-classifier to generate the final prediction. An experimental evaluation of an English vocabulary-based document dataset demonstrates that the proposed ensemble achieves higher accuracy, F1-score, and robustness compared to baseline deep learning and traditional machine learning methods. These results underscore the effectiveness of hybrid Transformer-driven ensembles in advancing document classification tasks.