DFIND OUTLIERS IN HIGH DIMENSION FULL

See: ' Fractional norms and quasinorms do not help to overcome the curse of dimensionality' by Mirkes, Allohibi, & Gorban (2020). However, later research has concluded against fractional norms. They produce some results which demonstrate that these 'fractional norms' exhibit the property of increasing the contrast between farthest and nearest points. The authors of the 'Surprising Behavior' paper then propose using $L_k$ norms with $k<1$. This means that the $L_1$ distance metric (Manhattan distance metric) is the most preferable for high dimensional applications, followed by the Euclidean metric ( $L_2$). Thus, for a given problem with a fixed (high) value for the dimensionality $d$, it may be preferable to use lower values of $k$. This provides considerable evidence that the meaningfulness of the $L_k$ norm worsens faster within increasing dimensionality for higher values of $k$. More specifically, we show that the relative contrasts of the distances to a query point depend heavily on the $L_k$ metric used. In this paper we provide some surprising theoretical and experimental results in analyzing the dependency of the $L_k$ norm on the value of $k$. Many high-dimensional indexing structures and algorithms use the uclidean distance metric as a natural extension of its traditional use in two- or three-dimensional spatial applications. In such cases, even the concept of proximity may not be meaningful from a qualitative perspective: a problem which is even more fundamental than the performance degradation of high dimensional algorithms. In such a case, the nearest neighbor problem becomes ill defined, since the contrast between the distances to diferent data points does not exist. It has been argued in that under certain reasonable assumptions on the data distribution, the ratio of the distances of the nearest and farthest neighbors to a given target in high dimensional space is almost 1 for a wide variety of data distributions and distance functions. Keim, ' On the Surprising Behavior of Distance Metrics in High Dimensional Space':

The premise of nearest neighbor search is that 'closer' points are more relevant than 'farther' points, but if all points are essentially uniformly distant from each other, the distinction is meaningless.įrom Charu C. This phenomenon can be observed for wide variety of distance metrics, but it is more pronounced for the Euclidean metric than, say, Manhattan distance metric. the points essentially become uniformly distant from each other. But in high dimensions, a curious phenomenon arises: the ratio between the nearest and farthest points approaches 1, i.e.

DFIND OUTLIERS IN HIGH DIMENSION FULL

The article is also full of many additional pearls of wisdom for machine learning.Īnother application, beyond machine learning, is nearest neighbor search: given an observation of interest, find its nearest neighbors (in the sense that these are the points with the smallest distance from the query point). This is bad news for machine learning, where shapes of one type are often approximated by shapes of another. And if we approximate a hypersphere by inscribing it in a hypercube, in high dimensions almost all the volume of the hypercube is outside the hypersphere. If a constant number of examples is distributed uniformly in a high-dimensional hypercube, beyond some dimensionality most examples are closer to a face of the hypercube than to their nearest neighbor. In high dimensions, most of the mass of a multivariate Gaussian distribution is not near the mean, but in an increasingly distant “shell” around it and most of the volume of a high-dimensional orange is in the skin, not the pulp. Ur intuitions, which come from a three-dimensional world, often do not apply in high-dimensional ones. A great summary of non-intuitive results in higher dimensions comes from ' A Few Useful Things to Know about Machine Learning' by Pedro Domingos at the University of Washington:

Dfind outliers in high dimension