High Performance Distributed Data Processing Architecture for AI-Based Multivariate Arctic Geospatial Predictive Weather Modeling

This paper describes a high performance, distributed artificial intelligence based approach for Arctic focused multivariate geospatial feature prediction to enhance weather model accuracy and computational efficiency through distributed dataset preprocessing. The proposed approach was tested using an Alaska wide time series air temperature dataset spanning 25 years. The workflow partitions the dataset into spatio-temporally independent chunks, which are preprocessed and scored locally. Feature scores from each chunk are then aggregated using a weighted voting average, ensuring that features consistently important across multiple regions are prioritized while maintaining global consistency. This distributed preprocessing offers benefits such as reduced computation time, enhanced model accuracy, improved generalization of AI models, and a more robust representation and understanding of complex geospatial data. Eight feature selection techniques covering filter, wrapper, and embedded methods were evaluated to identify the most relevant predictors, and their computational performance compared across distributed partitions. Four AI models were selected from linear, nonlinear, time-series machine learning, and non-time-series deep learning categories to predict air temperature and trained using databases of diverse regions from Alaska. The impact of the eight different feature selection techniques on model performance was evaluated using MSE, sMAPE, KGE, MPE (%), and prediction time. Results show that distributed preprocessing not only substantially reduces preprocessing time but also produces equivalent or better predictive performance compared to serial preprocessing. By capturing localized features and applying weighted aggregation, the workflow ensures global feature consistency while preserving regional significance. This leads to improved model generalization across heterogeneous Arctic regions and more accurate representation of spatially diverse geophysical patterns. Methods such as Kendall+DT and Pearson+MI exhibit superior scalability with increasing dataset size and computational resources, confirming their effectiveness for large-scale Arctic datasets.