𝑴𝑪𝑪𝑷𝒆𝒓𝒇𝒐𝒓𝒎𝒂𝒏𝒄𝒆: Towards unbiased selection of machine learning classification models – Case studies on environmental QSAR endpoints

Industrialization and human activities have led to the accumulation of a wide array of organic chemicals in the environment, most of which exert significant toxic effects. This warrants the quick and efficient filling of toxicity data gaps for developing safer alternatives, which is often not possible by trial-and-error experimentation. Therefore, researchers have developed multiple in silico-based machine learning models to predict the toxicity of these chemicals towards various environmental endpoints. However, there exist some practical challenges in the unbiased selection of machine learning models among similar competing models. The present study proposes three novel metrics that account for model robustness, generalizability, and overall model performance (𝑀𝐶𝐶𝑅𝑜𝑏𝑢𝑠𝑡𝑛𝑒𝑠𝑠, 𝑀𝐶𝐶𝐺𝑒𝑛𝑒𝑟𝑎𝑙𝑖𝑧𝑎𝑏𝑖𝑙𝑖𝑡𝑦, and 𝑀𝐶𝐶𝑃𝑒𝑟𝑓𝑜𝑟𝑚𝑎𝑛𝑐𝑒) that can be used quantitatively for an unbiased identification of the best classification-based model. Three different binary classification environmental fate and toxicity datasets (small, comparatively large, and medium-sized) were considered to evaluate the potential of these metrics. Each dataset was randomly split into a modeling and an independent test set. The modeling set was further split into training and validation sets three times using rational and random splitting approaches. A total of 72 machine learning models were trained, crossvalidated, and evaluated on the validation sets. The proposed metrics were computed for each model. Models possessing the highest 𝑀𝐶𝐶𝑃𝑒𝑟𝑓𝑜𝑟𝑚𝑎𝑛𝑐𝑒 value and acceptable values of 𝑀𝐶𝐶𝑅𝑜𝑏𝑢𝑠𝑡𝑛𝑒𝑠𝑠 and 𝑀𝐶𝐶𝐺𝑒𝑛𝑒𝑟𝑎𝑙𝑖𝑧𝑎𝑏𝑖𝑙𝑖𝑡𝑦 were selected as the best ones. We also analyzed some of the failed cases based on the threshold values that we propose. The best models, when subjected to independent test set predictions, generate better predictions than most of the models in the same train-validation split. Therefore, this work provides simple guidance for an unbiased selection of classification-based machine learning models that are expected to generate reliable external predictions.