Data Dimensions

Of the data scientist&s;s day-to-day tasks, a common chore is determining which data matters. While this is of course a conundrum when data is not available, it is a great challenge when the data available is overwhelming. The dimensions of large data sets are its attributes which can be split into subsets. Intuitively, data dimensions can be understood as separate columns ( fields ) of a table, or the different data files representing different attributes of related sets of data. Apart from algorithm complexity and computing resources, data processing programs must deal with the complexities of the data itself, not only in terms of its consistency and redundancy, but also in terms of its dimensions. A statistical overlap of data dimensions and redundancy is known as multicollinearity . Unnecessary costs arise from processing data dimensions that do not add value to the model, such as when a model is overly fit to noise, a situation called overfitting . Additionally, generating estimates with irrelevant data may be costly, as when they form the erroneous basis of business decisions.