Verification of AI–based environmental forecasting systems: What can we do, what do we need to do, and what are the challenges?

Several institutions have released global medium–range meteorological forecasting models based on methods from machine learning, with training data provided by various reanalysis experiments. A proper and in-depth assessment of these models and the quality of their forecasts has yet to be carried out. Although in terms of simple and overall measures of skill such as mean square errors, AI-based forecasts clearly show very promising skill, we are just beginning to understand where and when these forecasts are useful and when they are not. Furthermore, while verification of meteorological forecasts has been subject to extensive (and still ongoing) research with a well established core methodology, it is not clear to what extent this methodology needs to be adapted or modified for AI–based models. Our paper aims to provide a vision on the verification of AI–based weather forecasts, identifying challenges, outlining important research questions, and laying the groundwork for a methodology to assess the quality of such forecasts.