Contextual Fusion Strategies for Multimodal GNN-Based Reasoning: Performance and Computational Trade-Offs

The integration of multimodal data is essential for mission-specific tasks requiring comprehensive reasoning across diverse information sources, especially in resource-constrained environments. MissionGNN, a hierarchical Graph Neural Network (GNN) framework, has shown promise in deploying complex task models on edge devices by leveraging knowledge graphs (KGs). However, optimizing the fusion of different modalities remains a key challenge, as it significantly impacts the trade-off between computational efficiency and performance accuracy. In this paper, we propose three novel knowledge graph-centric fusion strategies for multimodal video anomaly detection within a hierarchical GNN-based reasoning framework—KG-based early fusion, late fusion,andreasoning fusion—along withmodality-aware knowledge graph generationto enhance modality-specific reasoning. Extensive experiments on the XD-Violence dataset evaluate their trade-offs across performance, computational cost, energy consumption, and LLM-basedKGgeneration cost. Early knowledge fusion offers a balanced solution, achieving significant accuracy improvements while reducing energy consumption by over 70% per frame. Late knowledge fusion, using half-sized modality-aware KGs, further improves accuracy with a modest increase in computational costs. Reasoning fusion achieves the highest accuracy of 72.93% mAP, demonstrating state-of-the-art performance, though it incurs higher resource usage. These results provide valuable insights for designing efficient multimodal GNN-based reasoning systems. By exploring these trade-offs, our work offers a framework to guide practitioners in selecting fusion strategies that balance energy efficiency and task-specific accuracy, tailored to mission-critical applications and varying resource constraints.