Deep sets of random neighbors with attention: An architecture for spatio-temporal machine learning

The study and modeling of spatio-temporal processes are essential in various fields, including geostatistics and environmental sciences. The presence of temporal dependencies, seasonality and evolving patterns in the data generated by these applications requires careful consideration to build accurate and reliable models, which poses several challenges. In light of this, in this paper, we propose a novel deep learning architecture tailored for spatio-temporal prediction in situations where data is sparse and irregularly sampled across both spatial and temporal dimensions. Our model consists of scenarios where data collection stations or mobile sensors can disappear or be established at any given time and location. Our approach predicts the variable of interest by aggregating information from random sets of neighboring data points. It learns a latent neighborhood representation and then maps this latent vector to produce a real-valued prediction. The aggregation process utilizes an attention mechanism, enabling the model to identify which neighbors provide the most meaningful and unique information. In addition to point predictions, our architecture offers a measure of prediction uncertainty. We validated our proposal by experimentation where it outperformed baseline models across two datasets, considering the sampling biases associated with static and mobile sensors. Furthermore, our model's uncertainty map effectively estimates absolute error, providing valuable information for prediction-based decisions. Our method also addresses skewed datasets with underrepresented value ranges.