Hierarchical Recurrent Neural Networks with Bayesian Dropout for Long-Sequence Language Modeling Tasks

Read full text Summarise with VIRAT
Authors: Khaled Abuhmaidan, Taher M. Ghazal
Cited by: 0
References: 43

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Statistical and neural language models continue to struggle with modeling long-sequence dependencies in natural language. This is primarily due to challenges inherent in recurrent architectures, including vanishing gradients, overfitting, and limited generalization. This work presents a Hierarchical Recurrent Neural Network framework enhanced with Bayesian dropout (HRNN-BD) for resilient long-sequence language modeling applications. The proposed architecture arranges recurrent layers into hierarchical levels. The lower layers capture short-term lexical and grammatical relationships, while the higher layers encode longer, contextual patterns that span longer sequences. Bayesian dropout is employed as a principled regularization method to address overfitting and uncertainty estimation. It enables the model to approximate posterior distributions over recurrent weights, improving its ability to generalize when training data are limited or noisy. Evaluations performed on benchmark datasets with long-text corpora demonstrate that the HRNN with Bayesian dropout outperforms traditional LSTM and GRU baselines in terms of perplexity, sequence coherence, and robustness. Furthermore, the hierarchical design assists with memory constraints while still allowing for efficient modeling of vast entities. The results show that combining hierarchical temporal abstraction with Bayesian regularization can enhance the state of the art in long-sequence language modeling. This approach can support document generation, summarization, and automated translation.

Fields of Study & Subjects

Topic Modeling Natural Language Processing Techniques Speech Recognition and Synthesis

Publication details

Journal / source
Procedia Computer Science
Volume
275
Pages
584–593
Publication date
1 January 2026
Language
EN