How to Achieve the Intended Aim of Deep Clustering Now, without Deep Learning

Deep clustering (DC) is often quoted to have a key advantage over $k$-means clustering. Yet, this advantage is often demonstrated using image datasets only, and it is unclear whether it addresses the fundamental limitations of $k$-means clustering. Deep Embedded Clustering (DEC) learns a latent representation via an autoencoder and performs clustering based on a $k$-means-like procedure, while the optimization is conducted in an end-to-end manner. This paper investigates whether the deep-learned representation has enabled DEC to overcome the known fundamental limitations of $k$-means clustering, i.e., its inability to discover clusters of arbitrary shapes, varied sizes and densities. Our investigations on DEC have a wider implication on deep clustering methods in general. Notably, none of these methods exploit the underlying data distribution. We uncover that a non-deep learning approach achieves the intended aim of deep clustering by making use of distributional information of clusters in a dataset to effectively address these fundamental limitations.

💡 Research Summary

This paper critically examines whether deep clustering methods, specifically Deep Embedded Clustering (DEC) and its improved variant IDEC, truly overcome the well‑known limitations of k‑means clustering—namely, the inability to discover clusters of arbitrary shapes, varied sizes, and differing densities. The authors begin by highlighting a conceptual problem: most clustering definitions in the literature (Definition 1) focus only on “high intra‑cluster similarity and low inter‑cluster similarity,” without specifying the structural characteristics of the desired clusters. Such a vague definition encourages the design of algorithms that rely on point‑to‑point similarity measures, which inherently limits them to spherical, equally sized, and equally dense clusters—exactly the scenario where k‑means excels.

To address this, the paper proposes a more precise definition (Definition 2) that explicitly requires the discovery of clusters with arbitrary geometry, size, and density. Building on this, the authors formulate additional definitions (Definitions 3 and 4) that articulate the necessary properties of any latent representation used by centroid‑based clustering methods: the representation must map each complex input‑space cluster into a compact, centroid‑representable region in the latent space.

Armed with these definitions, the authors evaluate DEC and IDEC. Both methods learn a non‑linear mapping via a stacked autoencoder and then perform a k‑means‑like assignment in the latent space, optimizing a KL‑divergence loss between a soft assignment Q (computed with a Student‑t kernel) and an auxiliary target distribution P. IDEC additionally retains the decoder and jointly optimizes reconstruction loss to preserve local structure. However, empirical experiments on synthetic datasets designed to expose the classic k‑means weaknesses—2‑Crescents (non‑convex shapes), Diff‑Sizes (clusters of markedly different cardinalities), and A‑C (clusters with differing densities)—show that DEC/IDEC achieve Normalized Mutual Information (NMI) scores only between 0.41 and 0.56, essentially matching k‑means performance. Visualizations of the learned latent spaces reveal that the clusters are not transformed into well‑separated spherical blobs, violating Definition 3’s requirement.

The authors identify the root cause: despite sophisticated representation learning, the final clustering step still depends on a distance‑based centroid assignment, which cannot capture the underlying distributional structure of the data. Consequently, the latent space does not encode the necessary distributional information to satisfy the more ambitious clustering goal.

To overcome this, the paper introduces the “Cluster‑as‑Distribution” (CaD) paradigm. Instead of defining clusters via pairwise distances, CaD treats each cluster as an i.i.d. sample from an unknown probability distribution. Inter‑cluster similarity is measured using a distributional kernel K(PX, PY), where PX and PY are empirical distributions of two point sets. This approach aligns with recent Kernel Bounded Clustering (KBC), which re‑expresses the spectral clustering objective (minimizing cuts) as maximizing intra‑cluster self‑similarity via a distribution kernel, thereby eliminating the need for eigen‑decomposition. In experiments, KBC attains NMI scores of 0.92–1.00 across all synthetic datasets, fully resolving the arbitrary‑shape, size, and density challenges.

The paper’s contributions are threefold: (1) a conceptual critique of existing clustering definitions and their impact on algorithm design; (2) a thorough empirical and theoretical demonstration that DEC/IDEC inherit k‑means’ fundamental limitations; (3) the proposal and validation of a non‑deep, distribution‑based clustering framework that achieves the intended deep‑clustering aim without learning a latent representation. The findings suggest that future clustering research should prioritize explicit modeling of data distributions rather than relying solely on representation learning, even when deep neural networks are employed.