Sample Weight Averaging for Stable Prediction

The challenge of Out-of-Distribution (OOD) generalization poses a foundational concern for the application of machine learning algorithms to risk-sensitive areas. Inspired by traditional importance weighting and propensity weighting methods, prior approaches employ an independence-based sample reweighting procedure. They aim at decorrelating covariates to counteract the bias introduced by spurious correlations between unstable variables and the outcome, thus enhancing generalization and fulfilling stable prediction under covariate shift. Nonetheless, these methods are prone to an inflation of variance, primarily attributable to the reduced efficacy in utilizing training samples during the reweighting process. Existing remedies necessitate either environmental labels or substantially higher time costs along with supervised information. To mitigate this issue, we propose SAmple Weight Averaging (SAWA), a simple yet effective strategy that can be universally integrated into various sample reweighting algorithms to decrease variance and estimation error of model parameters, thus achieving stable prediction across environments. Experiments across synthetic and real-world datasets consistently underscore its superiority against covariate shift.