Cost-Effective and Noise-Resilient Empirical Performance Modeling for HPC Applications

Analyzing the performance of applications and systems is essential in high-performance computing. To ensure optimal resource usage, improve scalability, identify bottlenecks, and enhance energy efficiency, continuous performance analysis is required. Extra-P, an automated empirical performance modeling tool, goes beyond traditional analysis by leveraging small-scale performance measurements to create scalability models predicting key metrics, such as runtime, based on execution parameters (e.g., the number of processes). This allows developers to predict application performance at different scales and make data-driven decisions without expert knowledge. However, several challenges limit Extra-P's effectiveness: the high cost of empirical experiments, system noise affecting measurement accuracy, and the lack of support for applications using hardware accelerators like GPUs, crucial for deep learning and AI tasks. Given the rapid growth of deep learning in HPC, particularly for training and deploying neural networks, Extra-P’s lack of support for these workloads is a significant limitation. This dissertation introduces novel modeling techniques to address these challenges and expand Extra-P's usability. The first contribution addresses the high costs of empirical performance modeling. By introducing a modeling technique that operates with a sparse matrix of data points, we reduce the required empirical performance experiments per model parameter from an exponential to a linear number, in contrast to Extra-P's dense-matrix approach that depends on complete data. This innovation significantly enhances experimental design freedom. Leveraging this, we develop two cost-effective experiment design strategies: reinforcement learning to derive a generic strategy and Gaussian process regression to identify optimal strategies tailored to specific modeling problems. Our results demonstrate an average cost reduction of 85% while retaining 92% of model accuracy. The second contribution addresses Extra-P's sensitivity to system noise. We introduce an adaptive modeling approach that enhances noise resilience by leveraging deep neural networks to uncover the influence of numerical parameters, such as the number of processes or problem size, on performance in the presence of noisy measurements. Our results show that this approach improves model accuracy under high noise levels by up to 25% and enhances predictive power by approximately 15%. Additionally, we explore the use of hardware counter measurements to improve the accuracy of models for performance metrics heavily affected by noise, such as application runtime. To evaluate their suitability for performance modeling, we conduct an extensive analysis and classify the counters based on their noise resilience. The third contribution extends Extra-P's capabilities to distributed deep learning applications with a new framework, Extra-Deep, for performance modeling and scalability analysis. Like Extra-P, Extra-Deep uses small-scale performance experiments to generate models for metrics such as training time per epoch as a function of execution parameters, with support for both data and model parallelism. These models enable detailed analyses of training tasks, including performance, scalability, efficiency, and cost. Additionally, we introduce an efficient sampling strategy that reduces profiling time for the required empirical measurements by an average of 94.9%. Our experiments demonstrate that Extra-Deep achieves an average prediction accuracy of 93.6% compared to empirical results. Overall, this work demonstrates how the proposed techniques enhance Extra-P's capability for analyzing HPC application performance. By improving usability, accuracy, noise resilience, and versatility while reducing modeling costs, it strengthens Extra-P as a more powerful and efficient tool for HPC analysis.