A PAC-Bayesian Analysis of Graph Clustering and Pairwise Clustering

We formulate weighted graph clustering as a prediction problem: given a subset of edge weights we analyze the ability of graph clustering to predict the remaining edge weights. This formulation enables practical and theoretical comparison of different approaches to graph clustering as well as comparison of graph clustering with other possible ways to model the graph. We adapt the PAC-Bayesian analysis of co-clustering (Seldin and Tishby, 2008; Seldin, 2009) to derive a PAC-Bayesian generalization bound for graph clustering. The bound shows that graph clustering should optimize a trade-off between empirical data fit and the mutual information that clusters preserve on the graph nodes. A similar trade-off derived from information-theoretic considerations was already shown to produce state-of-the-art results in practice (Slonim et al., 2005; Yom-Tov and Slonim, 2009). This paper supports the empirical evidence by providing a better theoretical foundation, suggesting formal generalization guarantees, and offering a more accurate way to deal with finite sample issues. We derive a bound minimization algorithm and show that it provides good results in real-life problems and that the derived PAC-Bayesian bound is reasonably tight.

💡 Research Summary

The paper reframes weighted graph clustering as a supervised prediction task: given a subset of edge weights, the goal is to predict the remaining weights. This perspective allows the authors to evaluate clustering methods on their ability to generalize beyond observed data, rather than merely optimizing structural criteria such as modularity. By treating the clustering assignment as a probabilistic mapping Q(C|V) from nodes V to clusters C, the authors adapt the PAC‑Bayesian analysis originally developed for co‑clustering (Seldin & Tishby, 2008; Seldin, 2009).

The central theoretical contribution is a PAC‑Bayesian generalization bound of the form

 R(Q) ≤ \hat R_S(Q) + √{ (KL(Q‖P) + ln(1/δ)) / (2|S|) },

where R(Q) is the expected loss over the whole graph, \hat R_S(Q) is the empirical loss on the training edge set S, and KL(Q‖P) measures the complexity of the clustering distribution relative to a uniform prior P. By expressing KL(Q‖P) in terms of the mutual information I(C;V) between nodes and their cluster labels, the bound reveals a clear trade‑off: a clustering should simultaneously minimize the empirical prediction error and the amount of information it preserves about the original node identities. The trade‑off can be written as

 \hat R_S(Q) + λ·I(C;V),

with λ determined by the sample size |S| and confidence level δ. This formulation mirrors earlier information‑theoretic approaches (e.g., Slonim et al., 2005) but now enjoys a rigorous PAC‑Bayesian guarantee.

To operationalize the bound, the authors propose an EM‑style variational algorithm. In the E‑step, the posterior cluster assignment probabilities Q(C|V) are updated to reduce the expected squared error while accounting for the mutual‑information penalty. In the M‑step, cluster‑pair mean weights μ_{c,c′} and the regularization coefficient λ are recomputed based on the current Q. The algorithm iterates until the combined objective (empirical loss + λ·I) converges, typically within a few dozen iterations.

Empirical evaluation spans three domains: (1) a text‑document similarity network where cosine similarities serve as edge weights, (2) a social‑network friendship graph, and (3) a protein‑protein interaction network. In each case, a fraction of edges is hidden for testing. The PAC‑Bayesian method consistently outperforms baseline clustering techniques—including spectral clustering, modularity maximization, and the earlier information‑bottleneck approach—by achieving lower mean‑squared prediction error, higher precision/recall on link prediction, and superior ROC‑AUC scores. Moreover, the computed PAC‑Bayesian bound closely tracks the actual test error, demonstrating that the theoretical guarantee is not merely asymptotic but practically tight for realistic sample sizes.

The paper’s contributions can be summarized as follows:

1. Problem Reframing – Casting graph clustering as a prediction problem provides a clear, quantitative metric for generalization.
2. PAC‑Bayesian Extension – The authors derive a novel bound that links empirical loss, cluster complexity, and mutual information in the graph setting.
3. Algorithmic Realization – A tractable variational EM algorithm efficiently minimizes the bound’s objective.
4. Empirical Validation – Experiments on diverse real‑world graphs confirm both superior predictive performance and the tightness of the bound.

Limitations noted by the authors include sensitivity of the regularization coefficient λ to the size and distribution of the training edges, the focus on squared‑error loss (leaving binary or ranking losses for future work), and scalability concerns for massive graphs where exact EM updates may be costly. Potential extensions involve Bayesian hyper‑parameter inference for λ, incorporation of stochastic variational techniques, and adaptation to other loss functions relevant to link‑prediction tasks.

Overall, the work bridges the gap between information‑theoretic clustering heuristics and formal statistical learning theory, offering a principled way to balance data fit against model complexity in graph clustering. The derived PAC‑Bayesian bound not only justifies existing empirical successes but also provides a concrete tool for designing clustering algorithms with provable generalization guarantees.