The Remarkable Simplicity of Very High Dimensional Data: Application of Model-Based Clustering

Abstract

An ultrametric topology formalizes the notion of hierarchical structure. Anultrametric embedding, referred to here as ultrametricity, is implied by ahierarchical embedding. Such hierarchical structure can be global in the dataset, or local. By quantifying extent or degree of ultrametricity in a data set, we show that ultrametricity becomes pervasive as dimensionality and/or spatial sparsity increases. This leads us to assert that very high dimensional data are of simple structure. We exemplify this finding through a range of simulated data cases. We discuss also application to very high frequency time series segmentation and modeling.