BIG-PU: An evaluation metric for exploration based on preference elicitation in recommender systems

In recent years, recommender systems have been widely adopted in various applications. Such systems predict user preferences for items based on past behavior and attribute information, determining which items to present. However, if recommendations are repeatedly made based on limited data, users may be exposed only to a narrow selection of items. This can result in many items remaining undiscovered, preventing users from finding new interests and potentially leading to a less satisfying long-term experience. To address this, “Exploration” deliberately recommends items with uncertain user preferences, promoting discovery. While beneficial in the long term, exploration can negatively impact short-term user experience by suggesting items of lower immediate preference. Thus, accurately measuring exploration’s effects is crucial. Existing evaluation methods based on diversity, novelty, and serendipity face challenges as their appropriate definitions depend on user and item characteristics. This research introduces a novel approach by defining exploration from the system’s perspective using Bayesian Information Gain. We propose an evaluation metric that quantifies exploration based on how much the system improves its understanding of user preferences. This metric, relying on the uncertainty of estimated preference distributions, offers greater universality than conventional methods. Through experiments on artificial and real datasets, we demonstrate its effectiveness, providing a new, broadly applicable framework for assessing exploration in recommender systems. Implementations are available at: https://github.com/tishii2479/big-pu .