Nocturnal non-speech sound classification with multi-spectrogram feature fusion and an attention-based stacked hybrid convolutional bidirectional long short-term memory–vision transformer architecture

Nocturnal non-speech sounds encapsulate critical physiological and behavioral information, making them a valuable modality for non-invasive assessment of sleep quality. Despite this potential, existing approaches predominantly rely on single-view spectral features or shallow learning architectures, limiting their ability to generalize across diverse acoustic patterns. To overcome these limitations, this study proposes a hybrid deep learning architecture tailored for the classification of seven distinct nocturnal sound categories. The system employs a tri-branch design that independently processes Mel-frequency cepstral coefficients (MFCC), Mel-spectrogram, and constant-Q transform (CQT)-spectrogram representations. Each branch passes through a dedicated pipeline comprising convolutional neural networks (CNN), bidirectional long short-term memory (BiLSTM) layers, and attention-equipped vision transformers (ViT). This configuration facilitates hierarchical learning of local, temporal, and global contextual features. The softmax outputs of each branch are fused using a stacking ensemble strategy, with an XGBoost-based meta-classifier performing the final decision integration. A complementary weighted ensemble is also implemented for comparative evaluation. Experimental results on a publicly available seven-class non-speech sound dataset demonstrate the proposed model’s outstanding performance, achieving 99.71% accuracy under 10-fold cross-validation, along with consistently high precision, recall, and F1-scores across all classes. Comparative benchmarks show substantial improvements over existing state-of-the-art models, including CNNs, long short-term memory (LSTM) variants, classical machine learning approaches, and metaheuristic-based ensembles. Supporting analyses such as confidence score distributions and dimensionality reduction visualizations (principal component analysis (PCA), t-distributed stochastic neighbor embedding (t-SNE)) further validate the model’s robustness and discriminative power. These findings highlight the effectiveness of integrating multi-spectral representations, deep hierarchical modeling, and ensemble strategies for high-fidelity nocturnal non-speech sound classification.