Inferring Pilot Cognitive States Using a Multimodal Physiology-Based Graph Fusion Model

Accurate recognition of pilots' cognitive states is essential for intelligent cockpit systems operating in complex and high-load flight environments. Existing methods either rely on single-modal physiological signals or fail to fully exploit the intrinsic temporal, spatial, and cross-modal structures of multimodal data. To address these limitations, we propose a multimodal physiology-based graph fusion (MPGFusion) network, an end-to-end deep framework that explicitly models the hierarchical graph structure and spatiotemporal dependencies of multimodal physiological signals. Unlike conventional approaches that depend on hand-crafted features or independent modality-specific encoders, MPGFusion directly takes raw multimodal physiological time series as input and jointly captures first, multiscale temporal dynamics via parallel temporal kernels, second, intramodality spatial relationships through single-modal graphs, and third, intermodality heterogeneous interactions using a multimodal graph integrated with graph transformation network, graph convolutional network, and gated recurrent unit. This hierarchical graph-based design enables structured cross-modal feature interaction while preserving modality-specific characteristics. Experiments conducted on the CogPilot dataset demonstrate that MPGFusion significantly outperforms representative baselines, achieving 97.49% classification accuracy. Extensive ablation studies validate the contribution of each architectural component and highlight the complementary nature of multimodal physiological signals. The proposed framework provides a generalizable and effective solution for multimodal cognitive state recognition in aviation scenarios.