Tuned your MPI library? Now check the performance guidelines

The MPI standard provides the foundational building blocks for most parallel applications running on large-scale HPC architectures.Collective communication operations in MPI are critical components for the scalability of these applications. Most MPI librariesoffer several algorithms for a specific collective operation, and each library selects the algorithm to be used based on the numberof processes, the message size, and possibly other factors. Each collective algorithm may perform better in certain scenarios, andthus, selecting the most suitable algorithm for each use case is essential. However, even the best algorithm in a given MPI librarymay deliver suboptimal performance.Self-consistent MPI performance guidelines capture semantic relationships between different collective operations and exploitthese to express performance expectations that collectives should reasonably satisfy to be considered performance-consistent. Forcollective communication, such performance guidelines typically state that a specialized collective call should not be slower thanless specialized counterparts.In this article, we demonstrate how the consistency of MPI libraries with respect to performance guidelines can be analyzed.For this purpose, we present a tool that checks guideline compliance. For regular collective operations such as MPI_Bcast, thetool contains multiple emulated versions of the collective by composing less specialized operations. Then, for a specific number ofprocesses and message sizes, the tool experimentally assesses whether the algorithm selected by the MPI library is slower than itsemulated counterparts. If that is the case, a performance-guideline violation is detected. In a broader empirical study, we assess thecurrent state of performance consistency in MPI libraries on modern supercomputers.