Learning Payoffs While Routing in Skill-Based Queues

Motivated by applications in service systems, we consider queueing systems where each customer must be handled by a server with the right skill set. We focus on optimizing the routing of customers to servers in order to maximize the total payoff of customer-server matches. In addition, customer-server-dependent payoff parameters are assumed to be unknown a priori. We construct a machine learning algorithm that adaptively learns the payoff parameters while maximizing the total payoff and prove that it achieves polylogarithmic regret. Moreover, we show that the algorithm is asymptotically optimal up to logarithmic terms by deriving a regret lower bound. The algorithm leverages the basic feasible solutions of a static linear program as the action space. The regret analysis overcomes the complex interplay between queueing and learning by analyzing the convergence of the queue length process to its stationary behavior. We also demonstrate the performance of the algorithm numerically and have included an experiment with time-varying parameters highlighting the potential of the algorithm in nonstatic environments. Funding: This work is part of Valuable AI, a research collaboration between the Eindhoven University of Technology and the Koninklijke KPN N.V. Parts of this research have been funded by the IMPULS program of the Eindhoven Artificial Intelligence Systems Institute, and by Holland High Tech through the Top Consortium for Knowledge and Innovation program High Tech Systems and Materials, via the public–private partnership allowance scheme. Supplemental Material: The online appendix is available at https://doi.org/10.1287/stsy.2024.0095 .