Median K-flats for hybrid linear modeling with many outliers

We describe the Median K-Flats (MKF) algorithm, a simple online method for hybrid linear modeling, i.e., for approximating data by a mixture of flats. This algorithm simultaneously partitions the data into clusters while finding their corresponding best approximating l1 d-flats, so that the cumulative l1 error is minimized. The current implementation restricts d-flats to be d-dimensional linear subspaces. It requires a negligible amount of storage, and its complexity, when modeling data consisting of N points in D-dimensional Euclidean space with K d-dimensional linear subspaces, is of order O(n K d D+n d^2 D), where n is the number of iterations required for convergence (empirically on the order of 10^4). Since it is an online algorithm, data can be supplied to it incrementally and it can incrementally produce the corresponding output. The performance of the algorithm is carefully evaluated using synthetic and real data.

💡 Research Summary

The paper introduces Median K‑Flats (MKF), an online algorithm designed for hybrid linear modeling (HLM), which seeks to represent a data set as a mixture of multiple low‑dimensional linear subspaces (flats). Traditional K‑Flats algorithms minimize an ℓ₂ loss, making them sensitive to outliers and requiring repeated passes over the entire data set, which is both memory‑ and time‑intensive. MKF replaces the ℓ₂ criterion with an ℓ₁ loss, thereby achieving robustness to a large proportion of corrupted points while retaining computational efficiency.

The algorithm proceeds iteratively. In each iteration a new data point is assigned to the flat that yields the smallest ℓ₁ distance (the sum of absolute deviations) to the point. After assignment, the corresponding flat is updated by solving an ℓ₁‑PCA problem on the points currently assigned to it. The ℓ₁‑PCA step can be performed with existing fast approximations, so the update cost is modest. Because the update uses only the newly arrived point and the current flat’s statistics, MKF operates in a truly online fashion: it does not need to store the whole data matrix, and memory consumption stays at O(Kd), where K is the number of flats and d their intrinsic dimension.

Complexity analysis shows that a single iteration costs O(K d D) operations (D is the ambient dimension). Empirically the algorithm converges after roughly n ≈ 10⁴ iterations, leading to an overall time complexity of O(n K d D + n d² D). When d ≪ D, this is dramatically lower than batch methods that require O(N K d D) for N data points. The authors also discuss initialization: a probabilistic seeding similar to K‑means++ reduces the chance of poor local minima, though the non‑convex nature of the ℓ₁ objective still precludes a global optimality guarantee.

Experimental evaluation is thorough. Synthetic tests generate data from several d‑dimensional subspaces embedded in D‑dimensional space, contaminated with Gaussian noise and uniformly distributed outliers at rates ranging from 10 % to 50 %. MKF consistently achieves lower mean absolute reconstruction error and higher clustering purity than K‑Flats, Generalized PCA (GPCA), and RANSAC‑based approaches, with improvements of 15–30 % in error metrics. Real‑world experiments include face image collections with varying illumination, motion‑capture sequences for human activity recognition, and 3D point‑cloud scans of indoor scenes. In all cases, MKF maintains stable performance even when outlier fractions exceed 30 %, a regime where ℓ₂‑based methods deteriorate sharply.

Additional analyses explore sensitivity to the number of flats K and subspace dimension d. The algorithm tolerates modest over‑estimation of K because redundant flats tend to attract few points and effectively become inactive. Likewise, modest mis‑specification of d does not dramatically harm performance, thanks to the ℓ₁ loss’s inherent regularization effect.

The paper concludes that MKF offers a compelling combination of low memory footprint, fast online updates, and strong robustness to outliers, making it suitable for streaming or embedded applications such as real‑time vision, robotics navigation, and large‑scale sensor networks. Future work is outlined: extending the framework to nonlinear manifolds, developing automatic model‑order selection (both K and d), and exploiting GPU parallelism to further accelerate the ℓ₁‑PCA sub‑routine.