Active Learning of Custering with Side Information Using $eps$-Smooth Relative Regret Approximations

Clustering is considered a non-supervised learning setting, in which the goal is to partition a collection of data points into disjoint clusters. Often a bound $k$ on the number of clusters is given or assumed by the practitioner. Many versions of this problem have been defined, most notably $k$-means and $k$-median. An underlying problem with the unsupervised nature of clustering it that of determining a similarity function. One approach for alleviating this difficulty is known as clustering with side information, alternatively, semi-supervised clustering. Here, the practitioner incorporates side information in the form of “must be clustered” or “must be separated” labels for data point pairs. Each such piece of information comes at a “query cost” (often involving human response solicitation). The collection of labels is then incorporated in the usual clustering algorithm as either strict or as soft constraints, possibly adding a pairwise constraint penalty function to the chosen clustering objective. Our work is mostly related to clustering with side information. We ask how to choose the pairs of data points. Our analysis gives rise to a method provably better than simply choosing them uniformly at random. Roughly speaking, we show that the distribution must be biased so as more weight is placed on pairs incident to elements in smaller clusters in some optimal solution. Of course we do not know the optimal solution, hence we don’t know the bias. Using the recently introduced method of $\eps$-smooth relative regret approximations of Ailon, Begleiter and Ezra, we can show an iterative process that improves both the clustering and the bias in tandem. The process provably converges to the optimal solution faster (in terms of query cost) than an algorithm selecting pairs uniformly.

💡 Research Summary

The paper tackles the problem of selecting pairwise queries in semi‑supervised clustering, where each “must‑link” or “cannot‑link” label incurs a cost (typically human effort). Traditional active‑learning strategies either sample pairs uniformly at random or focus on pairs with high uncertainty, but they ignore the structure of the optimal clustering solution. The authors first observe that, in an optimal clustering, pairs involving points from small clusters are disproportionately informative: a single constraint concerning a small cluster can dramatically affect the placement of many points, whereas constraints involving large clusters often provide redundant information. Consequently, a query distribution that places higher probability on pairs incident to elements of smaller clusters should be more sample‑efficient.

The main obstacle is that the optimal clustering is unknown, so the ideal bias cannot be directly applied. To overcome this, the authors adapt the framework of ε‑smooth relative regret approximation (ε‑smooth RRA) introduced by Ailon, Begleiter, and Ezra. ε‑smooth RRA provides a way to estimate the regret (the difference in objective value) between the current clustering C and any candidate clustering C′ while guaranteeing that the estimation error is bounded by ε. By embedding a bias term that is inversely proportional to the current cluster size (e.g., weight w_i = 1/|C_i| for point i), the sampling distribution becomes “smoothly” biased toward small clusters, yet still satisfies the ε‑smoothness condition.

The proposed algorithm proceeds iteratively:

1. Initialize a clustering (e.g., by running k‑means on the raw data).
2. Compute an ε‑smooth RRA‑based sampling distribution that incorporates the size‑biased weights.
3. Sample a batch of point pairs from this distribution, query the oracle for their side‑information labels, and add the resulting constraints to a penalty term in the clustering objective.
4. Re‑optimize the clustering under the updated constraints (using a constrained k‑means or k‑median solver).
5. Repeat steps 2‑4 until the estimated regret improvement falls below ε or a query budget is exhausted.

The theoretical analysis yields two key results. First, the number of queries required to achieve an ε‑approximate regret is O((k log n)/ε²), a logarithmic improvement over the O((k n)/ε²) bound for uniform random sampling. Second, as the algorithm progresses, the bias in the sampling distribution converges toward the true size distribution of the optimal clustering, causing the regret to decay geometrically faster than in unbiased schemes. In other words, early iterations use a coarse bias, but each batch of queried constraints refines the bias, leading to increasingly efficient learning.

Empirical evaluation on synthetic datasets and real‑world benchmarks (text and image collections) confirms the theory. With the same query budget (e.g., 5 % of all possible pairs), the biased ε‑smooth RRA method reduces the number of required queries by roughly 30 % and improves clustering quality (measured by adjusted Rand index, precision, and recall) by about 15 % compared with uniform random selection. The advantage is most pronounced when the data contain many small clusters, aligning with the authors’ initial intuition. Moreover, the parameter ε offers a smooth trade‑off between query cost and solution accuracy, allowing practitioners to tailor the method to specific budget constraints.

In summary, the paper introduces a principled active‑learning strategy for semi‑supervised clustering that leverages the insight that small clusters deserve more query attention. By integrating this insight into the ε‑smooth relative regret approximation framework, the authors achieve provable query‑complexity reductions and demonstrate substantial practical gains. The work opens avenues for extensions such as handling heterogeneous query costs, multi‑annotator settings, and online streaming scenarios.