Evaluating Multiple AI/ML Models with Parallelized Jupyter Notebook Deployment in Kubernetes for Accelerated Arctic Geospatial Preprocessing and Predictive Visualization

This paper describes a high-performance, distributed AI-based approach for Arctic-focused multivariate geospatial feature prediction, enhancing model accuracy and computational efficiency through parallel processing and distributed dataset preprocessing. The proposed approach was tested using an Alaska wide time series air temperature dataset spanning 25 years. The use of parallelized high performance computing architecture including, but not limited to Kubernetes (K8s) clusters, have the capacity, when properly configured with suitable cyberinfrastructure such as in the authors’ Arctic Knowledge Based System (A-KBS), to significantly accelerate large scale data analysis. The preprocessing and training pipelines were implemented in the JupyterLab environment utilizing Ray.io and TensorFlow. By leveraging the Ray.io library and employing a data chunking technique, the dataset was distributed across multiple worker nodes, enabling parallelization of data preparation tasks and reducing pre-processing time. We investigated eight feature selection techniques across three categories: filter, wrapper, and embedded methods, and compared their computational performance to identify the most relevant features for accurate and efficient air temperature prediction. Four generalized AI models trained using databases from Alaska were selected from linear, nonlinear, time series machine learning, and non time-series deep learning categories to predict air temperature. The impact of the eight different feature selection techniques on model performance was evaluated using MSE, sMAPE, KGE, MPE (%), and prediction time. The results showed that the proposed workflow significantly reduces processing time for most feature selection techniques and prediction models, particularly for those with high pre-processing demands. Notably, Kendall+DT and Pearson+MI methods exhibit better scalability as the number of allocated resources and dataset size increases, proving their effectiveness in handling large size datasets.