On the effect of noise on fitting linear regression models

The effects of including noise predictors and noise observations when fitting linear regression models are examined. Empirical and theoretical results show that double descent occurs in both cases, albeit with contradictory implications: for noise predictors, complex models are often better than simple ones, whereas for noise observations, simple models are often better than complex ones. This contradiction is resolved by showing that double descent is driven not by model complexity itself, but by the implicit shrinkage induced by the inclusion of noise in the model. Noise predictors or observations are shown to shrink the regression coefficient estimators towards zero and to generate asymptotic behaviour in the test error, while the combination of these asymptotes and the condition number anomaly ensures the occurrence of double descent. It is also shown that including noise observations makes the (usually unbiased) ordinary least squares estimator biased and may require a negative ridge parameter to avoid over-shrinkage.