A machine learning approach using autoencoders to perform quality control on meteorological data

Abstract As the volume of meteorological observations continues to grow, automating the quality control (QC) process is essential for timely data delivery. This study evaluates the performance of three machine learning algorithms—autoencoder, variational autoencoder, and long short-term memory (LSTM) autoencoder—for detecting anomalies in air temperature data. Using expert-quality-controlled data as ground truth, all models demonstrated anomaly detection capability, with the LSTM outperforming others due to its ability to capture temporal patterns and minimize false positives. When applied to raw data, the LSTM achieved 99.6% accuracy in identifying valid observations and replicated 79% of manual flags, with only five false negatives and six false positives over a full year. Its sensitivity to subtle meteorological changes, such as those caused by rainfall or cloud cover, highlights its robustness. The LSTM’s performance using a three-day timestep, combined with basic QC checks in SaQC (System for Automated Quality Control), suggests a scalable and effective solution for automated QC at Met Éireann, with potential for expansion to include additional variables and multi-station generalization.