Multi-Modal 3D Semantic Segmentation With Capsule-Weighted Knowledge Distillation

Accurate 3D semantic segmentation of outdoor environments is a fundamental requirement for autonomous systems, yet it remains challenging owing to the heterogeneous nature of sensor data. LiDAR sensors provide precise geometric information but lack semantic richness, whereas RGB cameras capture detailed appearance cues but suffer from unreliable depth estimation. Existing multi-modal fusion approaches typically apply uniform fusion strategies across all spatial regions, implicitly assuming equal feature reliability, which limits their effectiveness in complex scenes. We address this limitation by proposing a confidence-aware fusion strategy that uses capsule network activations to weight the knowledge transfer between 2D and 3D representations. Our key observation is that capsule vector magnitudes naturally encode feature presence confidence, which we leverage to guide adaptive fusion without requiring additional networks or supervision. The framework processes RGB images through a capsule-based encoder and LiDAR point clouds through a sparse point-voxel encoder. A four-stage fusion module then combines these modalities: first extracting capsule confidence scores, then weighting 2D features accordingly, projecting them to 3D space, and finally performing confidence-guided knowledge distillation from the 3D domain to enhance 2D feature learning. Confidence-aware fusion consistently outperforms uniform baselines across all evaluated categories. Experiments on SemanticKITTI confirm this hypothesis, achieving 66.8% mIoU on the test set and 70.6% on the validation set, with a 2.0% improvement over baselines while maintaining computational efficiency with only 21.4M parameters. These results validate confidence-aware fusion as a simple yet effective approach for multi-modal 3D semantic segmentation in autonomous driving applications.