Investigating extreme performance variability of a chemistry code

Occasionally, scientific codes run inefficiently for mysterious reasons. We report a high impact performance anomaly for the parallel ORCA quantum chemistry code on the REANNZ High-Performance Computing (HPC) platform. Identical parallel jobs exhibited unpredictable execution times, sporadically showing severe degradation ranging from 40 to 140 times slower than baseline performance on the same hardware. This variability severely impacts research throughput and resource utilisation.While investigations are ongoing, we found that the performance difference was linked to how the Message Passing Interface (MPI) processes were mapped onto the underlying Non-Uniform Memory Access (NUMA) architecture. Specifically, optimal performance requires contiguous process layout within a single Core Complex Die (CCD).In this talk, we will detail the investigative steps taken to isolate this dramatic performance drop, presenting the specific job characteristics and the underlying hardware profile of the REANNZ HPC platform. We provide a hypothesis of what the problem is, and some simple practical solutions that can partially mitigate for this detrimental breakdown in performance.ABOUT THE AUTHORGeoffrey Weal did his PhD in Computational Chemistry at the University of Otago under the supervision of Assoc. Prof. Anna Garden. His PhD involved the study of algorithms for locating energetically favourable shapes of gold and copper nanoparticles, as well as the catalysis of copper nanoparticles for producing carbon-neutral fuels. He then took up two postdoctoral positions at Victoria University of Wellington and Kyoto University (Japan) where he created a machine learning algorithm for predicting the electronic properties of organic molecules for the production of next generation, lightweight solar cells. He briefly took on a position at Advemto, a start-up company focused on creating highly resolved flow cytometers, before beginning his role at REANNZ as a Research Support Specialist. In this role, Geoffrey supports researchers undertake their research into all sorts of computational-based research, with a particular interest in computational chemistry related research. -----For more information about the eResearch NZ / eRangahau Aotearoa conference, visit: https://eresearchnz.co.nz/