DiffVTC-Net: Visual-Tactile Counterfactual Cross-Modal Weight Allocation Fusion Network Based on Diffusion Models

Multimodal representations for object recognition are critical for intelligent robotics, as they enable comprehensive scene understanding beyond single-modality limitations. Among these, the fusion of visual and tactile modalities holds significant promise in enhancing recognition accuracy and robustness. Nevertheless, existing visual-tactile fusion methods generally merely employ concatenation and addition to combine features. They seldom concentrate on the balanced alignment and profound interaction between the two modalities. In this paper, a counterfactual cross-modal weight allocation visual-tactile fusion network grounded in the anisotropic diffusion model (DiffVTC-Net) is proposed. The integration of visual and tactile features is formulated as a deterministic anisotropic diffusion process based on partial differential equations (PDEs). In this process, tactile information provides local guidance to refine visual features. This approach effectively captures the local information complementarity between modalities, providing a mathematically rigorous fusion mechanism that is distinct from probabilistic or generative diffusion models. To prevent errors from uncertainty in fused features, we propose a counterfactual cross-modal weight allocation module. This module uses counterfactual attention maps to guide attention map generation for both modalities. It quantifies each modality’s prediction contribution. Based on this, the module dynamically assigns weights to the real attention maps. These weighted maps are combined with the fused features for the final prediction. We conducted experiments on four public datasets to verify our method’s effectiveness. Ablation studies further demonstrated its performance advantages. Our code can be accessed from the following link: https://github.com/Srail-hh/vtf.