A Survey of Multi-Armed Bandit Algorithms: From Theoretical Foundations to Modern Applications

The multi-armed bandit (MAB) problem provides a canonical formulation of sequential decision-making under uncertainty, capturing the exploration-exploitation trade-off. This survey charts the intellectual lineage of MAB algorithms, ranging from their statistical roots to contemporary use as part of modern artificial intelligence. We begin by laying the mathematical groundwork for the problem in statistical terms. Building on this, we survey algorithms for the classical stochastic setting, covering simple heuristics such as -greedy to principled approaches based on optimism (Upper Confidence Bound, UCB) and Bayesian inference (Thompson Sampling). We then discuss the contextual bandit as the tipping point in which side information is introduced to enable large-scale personalization, and conclude with recent trends in bandit research, including adaptations to address real-world constraints, non- stationary environments, and the new frontier of fairness considerations. Overall, this review aims to showcase the broad applicability of bandit algorithms to problems ranging from recommendation systems to clinical trials, and argue for the continuing relevance of bandit methods.