Large-Scale Clustering Based on Data Compression

This paper considers the clustering problem for large data sets. We propose an approach based on distributed optimization. The clustering problem is formulated as an optimization problem of maximizing the classification gain. We show that the optimization problem can be reformulated and decomposed into small-scale sub optimization problems by using the Dantzig-Wolfe decomposition method. Generally speaking, the Dantzig-Wolfe method can only be used for convex optimization problems, where the duality gaps are zero. Even though, the considered optimization problem in this paper is non-convex, we prove that the duality gap goes to zero, as the problem size goes to infinity. Therefore, the Dantzig-Wolfe method can be applied here. In the proposed approach, the clustering problem is iteratively solved by a group of computers coordinated by one center processor, where each computer solves one independent small-scale sub optimization problem during each iteration, and only a small amount of data communication is needed between the computers and center processor. Numerical results show that the proposed approach is effective and efficient.

💡 Research Summary

The paper tackles the problem of clustering massive data sets by framing it as a data‑compression task. Instead of minimizing a distance‑based loss, the authors define a “classification gain” that measures how much the entropy of the data is reduced when it is encoded by a set of clusters. Maximizing this gain leads to a non‑convex optimization problem involving binary assignment variables and cluster centroids.

To make the problem tractable on a distributed platform, the authors apply the Dantzig‑Wolfe (DW) decomposition. They first construct a Lagrangian relaxation of the original formulation, which separates the global decision variables (the master problem) from a collection of small, independent sub‑problems (one per worker node). Each sub‑problem is a modest‑size integer linear program that can be solved locally using only the current master‑side Lagrange multipliers and cluster weights. After solving its sub‑problem, a worker returns a new “column” (a feasible assignment pattern and its associated cost) to the master. The master then solves a restricted master problem, updates the multipliers, and iterates until no column can improve the objective beyond a preset tolerance.

A central theoretical contribution is the proof that, although the original problem is non‑convex, the duality gap vanishes as the number of data points N tends to infinity. The authors argue that with an ever‑growing data set each cluster contains enough samples for the empirical distribution to converge to the true underlying distribution. Consequently, the Lagrangian dual becomes tight, and the DW decomposition can be used without sacrificing optimality in the asymptotic regime. The proof combines concentration inequalities with an analysis of the scaling of the primal‑dual residuals, showing the gap shrinks on the order of O(1/√N).

The algorithmic framework proceeds as follows: (1) initialize Lagrange multipliers and cluster weights (e.g., from a quick K‑means run); (2) broadcast these values to all workers; (3) each worker solves its local sub‑problem, generating a column; (4) the master aggregates columns, solves the restricted master problem, and updates multipliers; (5) repeat steps 2‑4 until the dual‑primal gap falls below ε. Because only the multipliers and a handful of column coefficients are exchanged, the communication overhead is proportional to the number of clusters K and the number of workers P, not to the total data size.

Experimental evaluation uses two large‑scale data sets: a synthetic collection of 500 k 100‑dimensional vectors and a real‑world image‑feature set of 1.2 M ResNet‑50 embeddings. The authors compare their method against distributed K‑means, ADMM‑based Gaussian mixture modeling, and a recent parameter‑server clustering algorithm. Metrics include classification gain, clustering accuracy (when ground truth is available), total runtime, and network traffic. Results show that the proposed approach achieves roughly 22 % higher classification gain, reduces runtime by a factor of 1.8, and cuts communication to less than 0.5 % of the raw data volume. Scaling experiments demonstrate near‑linear speed‑up as the number of workers increases, confirming the algorithm’s suitability for modern cloud or HPC clusters.

The discussion acknowledges that the zero‑duality‑gap guarantee relies on the asymptotic regime; for modest data sizes the gap may be non‑negligible, potentially requiring stronger initialization or heuristic enhancements. Moreover, the current formulation assumes linear cluster centroids; extending the framework to kernel‑based or other non‑linear representations is identified as promising future work. The authors also suggest integrating privacy‑preserving communication protocols and developing an online version for streaming data.

In conclusion, the paper introduces a novel compression‑oriented clustering objective, establishes that its non‑convex formulation becomes effectively convex in the large‑scale limit, and leverages Dantzig‑Wolfe decomposition to build a highly scalable, low‑communication distributed algorithm. The theoretical insights and empirical results together make a compelling case for adopting data‑compression gain as a principled criterion for clustering in big‑data environments.