Enhancing lasso tactical demand forecasts using semantic information

Leading indicators have been shown to be useful in predicting demand. Nowadays, a large number of potentially interesting variables are available in open databases. This makes it challenging to select the predictively relevant indicators. Lasso regression has become a popular choice when handling many variables, however, its selection is sensitive to changes in the sample and in the presence of correlated variables. This can harm both its predictive performance and the trustworthiness of the forecasts. We consider various approaches to aid variable selection. Specifically, we consider clustering via various statistical based methods, semantic information using a Semantic Bidirectional Encoder Representations from Transformers, or meta-data, such as a popularity index of variables. The resulting groups of variables are evaluated for selection directly, using sequential lasso, or transformed into cluster profiles, or factors using principal component analysis. Using an empirical case, we evaluate the alternative options on their decision and predictive performance, and on the interpretability of the resulting models. Our analysis indicates that there are trade-offs between predictive and decision performance, and interpretability. Semantic information is found to benefit variable selection, although we do not identify a dominant approach. Finally, we emphasise the need for methodological advances for explainability in forecasting.HighlightsWe compare different groupings of variables to enhance selection with lasso.Groupings are based on statistical, semantic, and meta-data information.Additional dimensionality reduction by cluster profiles and factors is investigated.Semantic information leads to gains in forecast and decision metrics.Credibly explainable models should match or outperform non-explainable benchmarks.