More is Not Always Better: Toward General Cross-Modal Saliency Prediction for Immersive Communications

Extensive previous studies demonstrate that visual saliency prediction significantly reduces bandwidth consumption in immersive communications, which serve as one key application in 6G by providing multi-sensory interactive experiences. Unlike traditional visual saliency prediction models designed for specific scenarios (e.g., visual-only or visual-audio contexts), this paper proposes a general cross-modal saliency prediction method. The proposed method fully exploits inter-modal correlations, and importantly, explores whether incorporating increasing modalities always benefits visual saliency prediction. Specifically, we construct an open-source multi-modal saliency dataset containing audio, video, and haptic modalities. Next, we propose a cross-modal saliency prediction (CMSP) model, which is capable of leveraging various modal combinations to generate saliency maps. In particular, CMSP extracts correlations among multi-modal features across spatial-temporal and channel dimensions, enabling adaptive fusion among various numbers of modalities. The experimental results show that for visual saliency prediction, integrating additional modalities can improve performance, but it comes at the cost of increased computational complexity and inference time. Therefore, the selection of modalities should be based on the accuracy requirements and the available computational resources.