SAIH: A Scalable Evaluation Methodology for Understanding AI Performance Trends on HPC Systems

The rapid convergence of artificial intelligence (AI) and high-performance computing (HPC) has created a pressing need for systematic methodologies to evaluate how AI workloads scale and perform on large-scale computing systems. Traditional HPC benchmarks and AI performance metrics often fail to capture evolving performance trends as model complexity, dataset size, and system scale increase simultaneously. This paper proposes SAIH (Scalable AI-HPC Evaluation Methodology), a comprehensive and reproducible framework designed to analyze AI performance trends on HPC systems across multiple scaling dimensions. SAIH integrates model scaling, data scaling, and system scaling to measure key performance indicators such as throughput, achieved computational efficiency, utilization, and scaling speedup. Using representative synthetic workloads that reflect realistic HPC-AI execution patterns, the study demonstrates how SAIH reveals non-linear performance behavior, diminishing returns at large scales, and bottlenecks arising from communication and I/O overheads. Experiments spanning model complexity (10M to 1B parameters), dataset sizes (0.025 to 1.6 TB), and system scales (1 to 128 nodes) demonstrate significant performance trends: throughput scales from 10 TFLOPS to 220.9 TFLOPS with model complexity, while strong scaling efficiency degrades from 100% to 26.6% due to communication overhead. The findings highlight the importance of scalable evaluation in guiding system design, workload optimization, and future AI-HPC co-design strategies.