A multimodal fusion approach with temporal convolutional networks for bearing fault diagnosis

Abstract To address the problems of insufficient information in single-modality data, difficulties in extracting high-quality fault features, poor generalization capability, and low diagnostic accuracy in rolling bearing fault diagnosis models, a novel Butterfly Optimization Algorithm-enhanced Squeeze-and-Excitation Temporal Convolutional Network (BOA_SETCN) based multimodal fusion method for rolling bearing fault diagnosis is proposed in this paper. First, the raw vibration data was preprocessed where Continuous Wavelet Transform (CWT) was applied for performing time-frequency analysis on the original one-dimensional vibration signal, with corresponding two-dimensional time-frequency maps being generated. A multimodal fusion approach was adopted in this paper, with both the original one-dimensional vibration signals and two-dimensional time-frequency maps being utilized as dual-input modalities for the proposed BOA_SETCN model. Initially, one-dimensional convolutional layers were employed by the framework to extract preliminary features from input signals. Subsequently, deep temporal features across multiple time steps were extracted by the Temporal Convolutional Network (TCN) architecture, which incorporates causal convolutions, dilated convolutions, and residual connections. A Squeeze-and-Excitation (SE) block was then introduced for the adaptive recalibration of channel-wise feature weights, thereby enhancing the feature representation capability. Finally, the Butterfly Optimization algorithm (BOA) was implemented to optimize critical TCN hyperparameters, including filter numbers, kernel sizes, and stride lengths. Experimental results demonstrated 99.44% accuracy on the Case Western Reserve University (CWRU) bearing dataset and 95.98% accuracy on the Southeast University (SEU) bearing dataset. The experimental results prove that the proposed model achieves higher accuracy compared to other methods, effectively improves fault diagnosis accuracy, and successfully completes the classification and identification of bearing fault types.