Bayesian Methodology for Adaptive Sparsity and Shrinkage in Regression

Bayesian and frequentist regularization techniques are commonly employed for estimating regression models from data. However, most existing methods require practitioners to pre-specify whether the target model is dense or sparse. A recent proposal, the GLP prior distribution, addresses this limitation by bridging the gap between sparse and dense model regimes in a data-driven manner. Despite its flexibility, the GLP prior incurs high computational costs due to matrix inversions and discretization-based sampling. This paper introduces a novel prior distribution, ACSS, designed to induce negative dependence between sparsity and shrinkage in regression models. Similar to the GLP prior, ACSS effectively estimates regression models across a wide spectrum of sparsity levels, from extremely sparse to fully dense. However, the ACSS distribution offers a significant computational advantage. Its Gibbs sampling strategy eliminates the need for matrix inversions and relies on sampling from standard univariate distributions, making it computationally efficient. Extensive experiments on synthetic and real data sets demonstrate that ACSS-based estimation is orders of magnitude faster than GLP. Moreover, in most cases, it also achieves superior estimation and predictive accuracy, underscoring its practical utility and effectiveness.