Wukong's Mind: Bridging Emotion and Behavior for Human-Centric Complex Video Understanding

Despite the remarkable progress in long video understanding via Multimodal Large Language Models (MLLMs), critical challenges persist in human-centric complex video understanding and reasoning. Existing benchmarks and video representation models predominantly focus on physical spatiotemporal dynamics, emphasizing perceptual-level analysis of human-object interactions through spatial details and temporal continuity. However, they overlook the deep fusion and driving relationships between human emotions and behaviors from an event semantic perspective, failing to bridge perception and cognition or provide cognitive-level causal reasoning in the semantic space. Inspired by Theory of Mind (ToM) and human cognitive processes in interpreting complex events, we bridge the gap between emotions and behaviors by proposing a novel framework, Wukong's Mind, with two key innovations: (i) Breaking through traditional event representation paradigms in spatiotemporal fields, we construct a structured event graph model in the event semantic space, enabling cross-spatiotemporal event understanding at cognitive-level. (ii) To evaluate humanlike reasoning capabilities, we introduce Wukong-QA, a benchmark constructed through an event-aware perceptionto-cognition annotation strategy, designed to systematically develop cognitive-level 5W1H question-answering types. Evaluated against dominant open-source MLLMs on Wukong-QA, our approach, Wukong's Mind, achieves an overall performance improvement over existing baselines. To our knowledge, this is the first attempt to explicitly bridge emotions and behaviors for cognitive-level humancentric complex video understanding, paving the way for more human-aligned video intelligence.