Surprising Geometrical Properties of High-Dimension Low-Sample Size Data with Devastating Consequences for Data Analysis

The advent of modern technology, permitting the measurement of thousands of variables simultaneously, has given rise to floods of data characterized by many large or even huge datasets. This new paradigm presents extraordinary challenges to data analysis and the question arises: how can conventional data analysis methods, devised for moderate or small datasets, cope with the complexities of modern data? The case of high-dimension low-sample size data is particularly revealing of some of the drawbacks. We look at the case where the number of variables measured in an object is at least the number of observed objects and conclude that (under the further assumptions that the data are observations from continuous random variables and that linear combinations of the variables are meaningful operations) this configuration leads to geometrical and mathematical oddities and is an insurmountable barrier for the direct application of traditional methodologies. If scientists are going to base their conclusions on high-dimension low-sample size data, ignoring fundamental mathematical results arrived at in this paper and blindly use software to analyze data, the results of their analyses may not be trustful, and the findings of their experiments may never be validated. That is why new methods together with the wise use of traditional approaches are essential to progress safely through the present reality.