itwinai: A Python Toolkit for Scalable Scientific Machine Learning on HPC Systems

Read full text
Authors: Matteo Bunino, Jarl Sondre Sæther, Linus Maximilian Eickhoff, Anna Elisa Lappe, Kalliopi Tsolaki, Killian Verder, Henry Mutegeki, Roman Machacek, Maria Girone, Oleksandr Krochak, Mario Rüttgers, Rakesh Sarma
+1 more Andreas Lintermann
Cited by: 0
References: 7

Abstract

The integration of Artificial Intelligence (AI) into scientific research has expanded significantly over the past decade, driven by the availability of large-scale datasets and Graphics Processing Units (GPUs), in particular at High Performance Computing (HPC) sites.However, many researchers face significant barriers when deploying AI workflows on HPC systems, as their heterogeneous nature forces scientists to focus on low-level implementation details rather than on their core research.At the same time, the researchers often lack specialized HPC/AI knowledge to implement their workflows efficiently.To address this, we present itwinai, a Python library that simplifies scalable AI on HPC.Its modular architecture and standard interface allow users to scale workloads efficiently from laptops to supercomputers, reducing implementation overhead and improving resource usage. Statement of needIntegrating machine learning into scientific workflows on HPC systems remains complex.Researchers must often invest substantial effort to configure distributed training, manage hyperparameter optimization, and analyze performance, while adapting to varied system architectures.itwinai is a Python library that streamlines this process by providing a unified framework for scalable AI workflows.It offers consistent interfaces for distributed training, supports large-scale hyperparameter optimization, and includes tools for profiling and code scalability analysis.

Fields of Study & Subjects

Scientific Computing and Data Management Parallel Computing and Optimization Techniques Distributed and Parallel Computing Systems

Publication details

Journal / source
The Journal of Open Source Software
Volume
11
Issue
117
Pages
9409–9409
Publication date
20 January 2026