Resampling

The development of models for estimation requires testing. Generally, this means splitting the original data source into separate data sets, one for training and one for testing. Using testing data to evaluate a model based on separate training data is known as cross validation . The cross validation process may be repeated for multiple testing sets with the results averaged for a single estimation. For training and test data sets to be relevant, the data must be randomly drawn from the same population or data stream. If the sample is too small or the selections are not sufficiently random, then the training and test sets may be too different and cross validation will not be relevant. In addition to covering random sampling for obtaining training and test data, this chapter discusses a method for comparing the means of two samples to check for randomness. This chapter also discusses frequency graphs, running averages, and calculating errors in estimates.