MM-NO: Learning Physical Operators from Heterogeneous Data Via Cross-Modal Attention Fusion

We introduce the Multi-Modal Neural Operator (MM-NO), a framework that pioneers the integration of multi-modal data fusion with operator learning to address the limitations of costly traditional solvers and unimodal data-driven models. Our approach is built on the insight that heterogeneous sources, such as sparse sensor readings and dense field data, contain complementary physical information. MM-NO leverages this through a cross-modal attention mechanism that fuses a Transformer encoder for sparse data with a Fourier Neural Operator (FNO) backbone. Evaluated on canonical systems like the Navier-Stokes and Shallow Water equations, MM-NO outperforms state-of-the-art baselines, reducing prediction error by up to 28%. Critically, it retains superior accuracy even when trained with 80% less dense field data. The model also exhibits remarkable zero-shot generalization to unseen physical parameters and achieves inference speeds over 1500× faster than traditional solvers. This work establishes a new paradigm for building scientific surrogates from multi-source, real-world data.