Research on multimodal feature alignment and semantic embedding model for cross-modal music retrieval

To address the challenges of feature heterogeneity and semantic gaps in cross-modal music retrieval, this paper proposes a multi-modal feature alignment and semantic embedding model, named the Hierarchical Attention and Contrastive Learning-based Multi-modal Semantic Alignment Network (HMSEAN). The model achieves dynamic alignment from local to global through a hierarchical attention mechanism, enhancing semantic matching for key music segments. It also constructs a shared semantic space fused with multi-level semantic information to improve the model's generalization capability. Specifically, the text branch uses pre-trained BERT, the audio branch extracts features through convolution layer and Bi-LSTM, and the image branch uses ResNet model. Cross-modal hierarchical attention alignment (CHAA) module realizes fine-grained semantic alignment through local segment alignment strategy, while Global Modal Alignment (GMA) module enhances cross-modal semantic alignment effect by cross-attention mechanism. In addition, by comparing the learning loss and the auxiliary semantic prediction loss, the shared semantic space is further optimized and the semantic understanding ability is improved. The experimental results on MusicCaps and MUSIC-AVQA data sets show that HMSEAN model is significantly superior to the existing methods in cross-modal retrieval tasks, and the indicators of R@1, R@5 and R@10 are all superior to the baseline model, which are 31.2%, 51.8%, 62.4% (text → audio) and 26.5%, 47.3% and 58.9 respectively. Ablation experiment and parameter sensitivity analysis further verify the effectiveness and parameter robustness of each module of the model.