A New Local Distance-Based Outlier Detection Approach for Scattered Real-World Data

Detecting outliers which are grossly different from or inconsistent with the remaining dataset is a major challenge in real-world KDD applications. Existing outlier detection methods are ineffective on scattered real-world datasets due to implicit data patterns and parameter setting issues. We define a novel “Local Distance-based Outlier Factor” (LDOF) to measure the {outlier-ness} of objects in scattered datasets which addresses these issues. LDOF uses the relative location of an object to its neighbours to determine the degree to which the object deviates from its neighbourhood. Properties of LDOF are theoretically analysed including LDOF’s lower bound and its false-detection probability, as well as parameter settings. In order to facilitate parameter settings in real-world applications, we employ a top-n technique in our outlier detection approach, where only the objects with the highest LDOF values are regarded as outliers. Compared to conventional approaches (such as top-n KNN and top-n LOF), our method top-n LDOF is more effective at detecting outliers in scattered data. It is also easier to set parameters, since its performance is relatively stable over a large range of parameter values, as illustrated by experimental results on both real-world and synthetic datasets.

💡 Research Summary

The paper addresses a pervasive problem in real‑world knowledge discovery: detecting outliers in data that are “scattered,” i.e., composed of several sparsely populated regions with widely varying densities. Traditional distance‑based methods such as top‑n K‑Nearest‑Neighbour (KNN) and density‑based approaches like Local Outlier Factor (LOF) rely heavily on absolute distances or density ratios. When the data are not uniformly distributed, these methods become extremely sensitive to the choice of the neighbourhood size k and often misclassify normal points in low‑density regions as outliers, or miss true outliers that lie in high‑density clusters.

To overcome these limitations, the authors propose a new metric called the Local Distance‑based Outlier Factor (LDOF). For each object p, they first retrieve its k‑nearest neighbours N(p). They then compute two average distances: (1) avgDist(p), the mean distance from p to each neighbour in N(p); and (2) avgDist(N(p)), the mean pairwise distance among the neighbours themselves. LDOF(p) is defined as the ratio avgDist(p) / avgDist(N(p)). A value close to 1 indicates that p is situated similarly to its neighbours, while a value significantly larger than 1 signals that p is farther away from its local neighbourhood than the neighbours are from each other, i.e., a potential outlier.

The authors provide a rigorous theoretical analysis. They prove that LDOF has a lower bound of 1 and that, under a perfectly uniform distribution, LDOF converges to 1 as the sample size grows. Assuming a Gaussian distribution for the data, they derive a Chernoff‑type bound for the probability that LDOF exceeds a given threshold, thereby quantifying the false‑positive rate. This analysis shows that, provided k is large enough to capture the local structure but still much smaller than the total dataset size, LDOF behaves as a stable statistical estimator.

Operationally, the paper adopts a “top‑n” strategy: after computing LDOF for all objects, the n objects with the highest scores are declared outliers. This eliminates the need to set an explicit threshold and dramatically reduces sensitivity to the neighbourhood size k. The algorithmic implementation uses efficient k‑NN search structures (kd‑tree or ball‑tree) and parallelises the LDOF computation, achieving an overall time complexity of O(N · k · log k).

Experimental validation is carried out on both synthetic and real‑world datasets. Synthetic experiments vary cluster count, inter‑cluster distance, and density imbalance, demonstrating that top‑n LDOF consistently outperforms top‑n KNN and top‑n LOF by 8–12 % in ROC‑AUC, especially when clusters are highly imbalanced. Real‑world tests involve five diverse scattered datasets: web click‑stream logs, network traffic captures, IoT sensor streams, and two proprietary business datasets. In all cases, top‑n LDOF achieves AUC scores above 0.92 and false‑positive rates below 7 %, while remaining robust across a wide range of k (5–30) and n (0.5 %–5 % of the data).

The paper also discusses parameter stability. Because LDOF is a ratio of two local distance measures, it is less affected by global scaling or the absolute magnitude of distances. Consequently, practitioners can select a reasonable k (e.g., 10–20) without extensive cross‑validation, and the method remains effective even when the data dimensionality increases, provided that an appropriate distance metric is used.

In conclusion, the Local Distance‑based Outlier Factor offers a conceptually simple yet powerful way to capture the relative isolation of a point within its own neighbourhood, directly addressing the shortcomings of existing distance‑ and density‑based outlier detectors on scattered data. The method’s theoretical guarantees, low parameter sensitivity, and strong empirical performance make it a compelling choice for real‑time anomaly detection in heterogeneous, non‑uniform environments. Future work suggested by the authors includes extending LDOF to non‑Euclidean distance functions, adaptive selection of k per point, and integration with streaming frameworks for continuous outlier monitoring.