When Vision-Language Model (VLM) Meets Beam Prediction: A Multimodal Contrastive Learning Framework

As the real propagation environment becomes increasingly complex and dynamic, millimeter wave beam prediction faces significant challenges. However, the powerful cross-modal representation capability of vision-language model (VLM) provides a promising approach. The traditional methods that rely on real-time channel state information are computationally expensive and often fail to maintain accuracy in such environments. In this paper, we present a VLM-driven multimodal beam prediction framework that integrates multimodal data via modality-specific encoders. To enforce cross-modal consistency, we adopt a sequential contrastive pre-training strategy to align visual features with text prompts for semantic injection, followed by alignment with light detection and ranging. We use location information as text prompts and connect it to the text encoder to introduce the language modality, which further improves cross-modal consistency. Experiments on theDeepSense-6Gdataset show that our VLM backbone provides additional semantic grounding. Compared with existing methods, the proposed framework achieves an overall distance-based accuracy score of 0.9034, corresponding to 1.65% average improvement.