Towards Understanding and Avoiding Limitations of Convolutions on Graphs

While message-passing neural networks (MPNNs) have shown promising results, their real-world impact remains limited. Although various limitations have been identified, their theoretical foundations remain poorly understood, leading to fragmented research efforts. In this thesis, we provide an in-depth theoretical analysis and identify several key properties limiting the performance of MPNNs. Building on these findings, we propose several frameworks that address these shortcomings. We identify two properties exhibited by many MPNNs: shared component amplification (SCA), where each message-passing iteration amplifies the same components across all feature channels, and component dominance (CD), where a single component gets increasingly amplified as more message-passing steps are applied. These properties lead to the observable phenomenon of rank collapse of node representations, which generalizes the established over-smoothing phenomenon. By generalizing and decomposing over-smoothing, we enable a deeper understanding of MPNNs, more targeted solutions, and more precise communication within the field. To avoid SCA, we show that utilizing multiple computational graphs or edge relations is necessary. Our multi-relational split (MRS) framework transforms any existing MPNN into one that leverages multiple edge relations. Additionally, we introduce the spectral graph convolution for multiple feature channels (MIMO-GC), which naturally uses multiple computational graphs. A localized variant, LMGC, approximates the MIMO-GC while inheriting its beneficial properties. To address CD, we demonstrate a close connection between MPNNs and the PageRank algorithm. Based on personalized PageRank, we propose a variant of MPNNs that allows for infinitely many message-passing iterations, while preserving initial node features. Collectively, these results deepen the theoretical understanding of MPNNs.

💡 Research Summary

**
This paper provides a comprehensive theoretical analysis of two fundamental limitations that affect the performance of message‑passing neural networks (MPNNs) on graphs: Shared Component Amplification (SCA) and Component Dominance (CD). Both phenomena lead to a progressive loss of rank in node representations, a generalization of the well‑known over‑smoothing problem.

Shared Component Amplification (SCA) occurs when each message‑passing iteration applies the same linear operator (typically a graph Laplacian or a normalized adjacency matrix) to all feature channels. In spectral terms, the same eigenvectors are repeatedly amplified by the same eigenvalues across every channel. As the number of layers grows, the representation collapses onto a low‑dimensional subspace spanned by a few dominant eigenvectors, causing the rank of the node‑feature matrix to shrink dramatically.

Component Dominance (CD) describes the situation where one eigenvalue (usually the largest) is close to or exceeds one, so its associated eigenvector grows much faster than the others. Consequently, a single spectral component dominates the whole representation, making the embeddings converge to a direction that is largely independent of the original node features. CD intensifies the rank‑collapse caused by SCA and explains why deeper GNNs often become ineffective.

The authors formalize these effects by expressing a generic MPNN update as
(X^{(t+1)} = \Theta X^{(t)}),
where (\Theta) is the message‑passing matrix. By performing a spectral decomposition (\Theta = U \Lambda U^{\top}), they show that the evolution of each eigen‑component follows (\lambda_i^{t}). SCA is present whenever (\Theta) is identical for all channels, while CD appears when (\max_i |\lambda_i|) is close to one. This analysis reveals that many popular architectures—GCN, GraphSAGE, GAT, GIN, and their variants—are intrinsically vulnerable to both SCA and CD.

To mitigate SCA, the paper introduces two complementary constructions:

1. Multi‑Relational Split (MRS) – The original graph is decomposed into several edge‑type specific sub‑graphs. Each sub‑graph has its own message‑passing matrix, thus providing distinct spectra for different channels. By aggregating the results from all relations, the network avoids repeatedly amplifying the same components.

2. Spectral Multi‑Input Multi‑Output Graph Convolution (MIMO‑GC) – This operator processes multiple feature channels simultaneously with a set of distinct spectral filters. Each filter corresponds to a different Laplacian (or a different polynomial of the Laplacian), effectively realizing a multi‑relational computation without explicitly constructing separate graphs. A localized approximation, LMGC, reduces computational cost by restricting the filters to a small neighbourhood while preserving the diversity of spectra.

To address CD, the authors exploit the close relationship between MPNNs and the Personalized PageRank (PPR) algorithm. PPR introduces a teleportation (or damping) factor (\alpha \in (0,1)) that guarantees all eigenvalues of the propagation matrix lie strictly inside the unit circle. This enables an infinite‑step message‑passing scheme that never loses the initial features. The paper proposes a PPR‑MPNN variant, where the update rule becomes
(X^{(\infty)} = (1-\alpha)(I - \alpha \Theta)^{-1} X^{(0)}).
Theoretical analysis shows that the rank of (X^{(\infty)}) remains full as long as the original feature matrix is full rank, thereby eliminating CD‑induced collapse.

Experimental validation is performed on several standard benchmarks (OGB‑Products, Cora, PubMed, and synthetic deep‑GNN tasks). The authors compare the proposed MRS, MIMO‑GC, LMGC, and PPR‑MPNN against baseline GCN, GraphSAGE, GAT, and recent deep GNN designs. Key findings include:

• Depth robustness – While conventional GNNs degrade sharply after 10–15 layers (often dropping below 10 % accuracy), the proposed methods maintain high performance even with 30–50 layers, achieving >70 % accuracy on OGB‑Products.
• Rank preservation – Singular‑value analysis shows that the node‑feature matrix retains a high effective rank throughout training for the new models, whereas baselines quickly collapse to rank‑1 or rank‑2.
• Computational efficiency – LMGC runs roughly twice as fast as the full MIMO‑GC with negligible loss in accuracy, making it suitable for large‑scale or real‑time applications.
• Ablation studies – Removing the multi‑relational split re‑introduces SCA, and disabling the PPR damping re‑creates CD, confirming the causal role of each component.

In summary, the paper makes three major contributions:

1. Theoretical clarification of SCA and CD as the underlying mechanisms of over‑smoothing and rank collapse in graph convolutions.
2. Algorithmic frameworks (MRS, MIMO‑GC/LMGC, and PPR‑MPNN) that systematically break the shared‑component amplification and component‑dominance patterns.
3. Empirical evidence that these frameworks enable deep graph neural networks to scale to many layers without sacrificing expressive power or computational practicality.

The work deepens our understanding of why many graph convolutional models fail in deep regimes and provides concrete, mathematically grounded tools to design next‑generation GNNs that are both deep and robust.