The hidden impact of data management on data quality traced using simulated Rare Earth Element pattern recognition

The data life cycle describes the stages of data acquisition, from collecting samples in the field, analysing, processing, publishing, and archiving data so that it might be reused in other contexts. Any errors occurring anywhere in the life cycle thus accumulate in the last stages, specifically data compilations, data repositories, data portals and databases. The presence of “bad” analyses in geochemical compilations is not a new phenomenon. Strategies for handling these bad apples usually involve various “cleaning” procedures to throw out samples. However, those strategies miss important questions to meaningfully remedy the phenomenon: Why do errors occur and how can we identify the type and source of error? Visually assessing Rare Earth Element (REE) patterns is commonly used as a “cleaning” procedure to check the quality of a rock or mineral sample. The REEs behave systematically in natural rocks/minerals due to their systematic atomic properties resulting in usually smooth normalized REE patterns (excluding well-known Eu and Ce anomalies). Utilizing the REEs, we deep dive into two of the largest public geochemical databases (GEOROC and PetDB) to answer the above posed questions. Our assessment of the REE data in the databases show that the vast fraction of odd looking, zig-zaggy and unnatural patterns track back to data handling and management errors more often than “bad” analyses in the analytical sense. These problems were prevalent in the original published papers too. Common handling errors include offset/switched columns or rows, rounding, normalization, decimal scale/units shifts and rounding. We are developing a Machine Learning framework using REE patterns that can predict if an error is present and the most likely cause. The training dataset contains the complete collection of samples with originally smooth REE patterns from GEOROC, on which we simulate the different data management errors. To truly capture the patterns, the feature design utilises the principles of compositional data analysis and pairwise log-ratios to capture both the shape of the REE pattern and neighbouring REE behaviour. The resulting model can be applied on any compositional dataset compilation or in other geochemical data repositories or databases that contain REE data. The insights of the model can thus focus the effort to improve geochemical dataset quality by locating and labelling the errors. However, it also highlights the need for a cultural change in the geochemical community to improve data management in the entire data life cycle, a prerequisite for unlocking any potential of Big Data in geochemistry.