AI-Driven Fusion of Multimodal Channel Sounder Data for RF Heatmap Segmentation

Integrated sensing and communications (ISAC) systems detect physical targets within the wireless channel while supporting data transmission. Direct use of measured radio-frequency (RF) heatmaps, rather than extracting multipath components from them, preserves physically meaningful propagation features often lost under sparsity assumptions. However, conventional (RF-only) sounders rely on motion cues to isolate targets in the heatmaps, limiting analysis to scenes with a single mobile target. We present a multimodal data-fusion framework that overcomes this limitation by incorporating lidar and camera systems into our channel sounder, from which digital twins of the scene targets are generated through deep learning algorithms. The RF, lidar, and camera systems are spatially co-registered and temporally synchronized, establishing voxel-level correspondence between the 3D digital twins and 3D RF heatmaps. This correspondence enables projection of the twins into the heatmaps, where they act as semantic masks for geometry-based target segmentation. Validation over 54000 channel acquisitions at 28.5GHz across diverse geometries demonstrates enhanced interpretability and target isolation. The results highlight the potential of multimodal fusion for AI-assisted channel modeling and ISAC in future 6G systems.