Statistical Issues Often Overlooked when Analyzing Astronomical Data

Abstract The main topics covered in this paper are (1) controlling significance levels when applying the same hypothesis test to many (possibly millions) of datasets; (2) dealing with the fact that for very large datasets hypotheses are rejected for trivially small departures from the null; (3) in the presence of noise, extreme values selected from samples for follow-up studies are often biased; (4) inference conducted on models fitted to data routinely underestimate the parameter standard errors if the selected model was informed by the observations; (5) obtaining overall least-mean-squared error estimates of a group of observations (e.g. a collection of star cluster masses); and (6) the effects of the violation of mathematical regularity conditions on model selection statistics.