Exascale-Ready Scientific Foundation Models via Multi-Dimensional Hybrid Parallelism and Topology-Aware Scheduling

The rapid emergence of scientific foundation models has created unprecedented opportunities for datadriven discovery across domains such as climate science, computational biology, and multi-physics simulations. Training such models efficiently requires careful consideration of data representation, parallelization strategy, and hardware topology. This paper presents a comprehensive framework for physics-informed neural operator training that integrates multi-dimensional hybrid parallelism with topology-aware scheduling. The proposed method incorporates physics-informed tokenization, multi-axis domain decomposition, and coordinated data, tensor, and pipeline parallelism to minimize synchronization overhead. Experimental validation on small-scale multi-GPU systems (2-8 GPUs) training Fourier Neural Operators for Burgers' equation demonstrates substantial improvements: 2.30× average throughput speedup, 15.5% GPU memory reduction, 57.9% faster convergence, and 81.1% improved physics constraint satisfaction. These results validate that co-optimizing physical data representation, model partitioning, and hardware-aware execution is essential for efficient scientific foundation model training. The framework achieves superlinear scaling with physics regularization benefits, providing a foundation for efficient exascale AI infrastructure.