Proposition 1 (Mitigating Over-smoothing with HistoGraph). *Let $`\mathbf{x}_v^{(l)} \in \mathbb{R}^D`$ denote the embedding of node $`v`$ at layer $`l`$ of a GNN. Suppose the GNN exhibits over-smoothing, i.e., there exists some layer $`L_0`$ sufficiently large such that for all layers $`l_1, l_2 > L_0`$ and all nodes $`u,v`$,

\begin{equation}
\label{eq:oversmooth-def}
\|\mathbf{x}_u^{(l_1)} - \mathbf{x}_v^{(l_2)}\| \rightarrow 0.
\end{equation}
```*

*Let <span class="smallcaps">HistoGraph</span> compute the final node
embedding as
``` math
\begin{equation}
h_v = \sum_{l=0}^{L-1} \alpha_l \mathbf{x}_v^{(l)},
\end{equation}

where $`\alpha_l`$ are learned attention weights. Then, for distinct nodes $`u`$ and $`v`$, there exists at least one layer $`l' \leq L_0`$ with $`\alpha_{l'} \neq 0`$ such that

\begin{equation}
\|h_u - h_v\| > 0.
\end{equation}
```*

*That is, <span class="smallcaps">HistoGraph</span> retains information
from early layers and mitigates over-smoothing.*

</div>

<figure id="fig:attention_and_embedding_evolution"
data-latex-placement="t">
<figure id="fig:layerwise_attention_scores">
<p><img src="/posts/2026/01/2026-01-03-190637-learning_from_historical_activations_in_graph_neur/combined_attention_visualization_IMDBB_64l.png"
alt="image" /> <span id="fig:layerwise_attention_scores"
data-label="fig:layerwise_attention_scores"></span></p>
</figure>
<figure id="fig:embedding_norm_diff">
<p><img src="/posts/2026/01/2026-01-03-190637-learning_from_historical_activations_in_graph_neur/combined_visualization_IMDBB_64l.png" alt="image" />
<span id="fig:embedding_norm_diff"
data-label="fig:embedding_norm_diff"></span></p>
</figure>
<figcaption>Visualizations on the <span class="smallcaps">imdb-b</span>
dataset with 64-layer <span class="smallcaps">HistoGraph</span>. (left)
Attention patterns across layers under different training regimes.
(right) Embedding evolution throughout training, measured by the normed
difference between final and intermediate representations.</figcaption>
</figure>

**<span class="smallcaps">HistoGraph</span>’s Layer-wise Attention is an
Adaptive Trajectory Filter.** <span id="prop:filter"
label="prop:filter"></span> We interpret
<span class="smallcaps">HistoGraph</span>’s layer-wise attention as an
Adaptive Trajectory Filter, which dynamically aggregates a node’s
embeddings across layers based on learned weights. Let
$`\{\mathbf{x}^{(l)}\}_{l=0}^{L-1} \subset \mathbb{R}^D`$ denote a
node’s embeddings at each layers. We define the aggregated embedding as:
``` math
\begin{equation}
\mathbf{h} = \sum_{l=0}^{L-1} \alpha_l \mathbf{x}^{(l)}, \quad \text{with} \quad \sum_l \alpha_l = 1.
\end{equation}

where $`\alpha_l`$ are learnable attention weights. Depending on $`\alpha_l`$, the aggregation implements: (i) a low-pass filter when $`\alpha_l=\tfrac{1}{L}`$ (uniform average); (ii) a high-pass filter when $`\alpha_l=\delta_{l,L-1}-\delta_{l,L-2}`$ (first difference); and (iii) a general FIR filter when $`\alpha_l`$ are learned. Consequently, layer-wise attention in HistoGraph treats the GNN’s historical activations as a sequence and learns flexible, task-driven filtering and aggregation for the final classifier. Figure 5 illustrates a case where GCN fails at high-pass filtering, whereas HistoGraph succeeds. The barbell graph — a symmetric clique joined by a single edge—creates a sharp gradient discontinuity, highlighting how the adaptive filtering of HistoGraph preserves such signals, unlike standard GCNs. Appendix 10 further analyzes the usefulness of node-wise attention in HistoGraph.

/>

/>

/>

/>

Experiments

In this section, we conduct an extensive set of experiments to demonstrate the effectiveness of HistoGraph as a graph pooling function. Our experiments seek to address the following questions:

1. Does HistoGraph consistently improve GNN performance over existing pooling functions across diverse domains?

2. Can HistoGraph be applied as a post-processing step to enhance the performance of pretrained GNNs?

3. What is the impact of each component of HistoGraph on performance?

Baselines. We compare HistoGraph against diverse baselines spanning graph representation and pooling. Message-passing GNNs: GCN and GIN with mean or sum aggregation . Set-level pooling: Set2Set . Node-dropping pooling: SortPool , SAGPool , TopKPool , ASAP . Clustering-based pooling: DiffPool , MinCutPool , HaarPool , StructPool . EdgePool  merges nodes along high-scoring edges. Attention-based global pooling: GMT . Additional models: SOPool , HAP , PAS , GMN , DKEPool , JKNet . On TUdatasets, we also include five kernel baselines: GK , RW , WL subtree , DGK , and AWE . An overview of baseline characteristics versus HistoGraph appears in Table [tab:pooling-methods].

Benchmarks. We use the OGB benchmark  and the widely used TUDatasets ; dataset statistics appear in [tab:ogb-stats,tab:tudata-stats] in Appendix 7. For OGB, we follow with 3 GCN layers; for TUDatasets, we adopt , typically using 5 GIN layers. For deeper variants, we keep the backbone and vary the number of layers. Hyperparameters are in Appendix 9.1. Additionally, we benchmark HistoGraph on several node-classification datasets spanning heterophilic and homophilic graphs (Table 4) and across varying GNN depths (Table 1). Further results appear in Appendix 12: feature-distance across layers for GCN and GCN with HistoGraph (Table 5), comparison to the GraphGPS baseline (Table 7), and link prediction (Table 6).

End-to-End Activation Aggregation with HistoGraph

We evaluate end-to-end activation aggregation with HistoGraph on graph-level benchmarks and node classification. We first report results on TUDatasets (Table [tab:turesults]), followed by OGB molecular property prediction (Table [tab:ogb_results]), and finally depth-scaled node classification (Table 1).

TUDatasets. On seven datasets  (imdb-b, imdb-m, mutag, ptc, proteins, rdt-b, nci1), HistoGraph attains state-of-the-art performance on 5 of 7: imdb-b 87.2%, imdb-m 61.9%, mutag 97.9%, proteins 97.8%, nci1 85.9%. It is marginally behind on ptc at 79.1% versus 79.6% for DKEPool. Relative to the second-best method, gains are substantial on proteins (+16.6%), imdb-b (+6.3%), and imdb-m (+5.6%). Although DKEPool slightly leads on ptc and rdt-b, the overall trend favors HistoGraph across diverse graph classification benchmarks.

OGB molecular property prediction. On four OGB datasets  (molhiv, moltox21, toxcast, molbbbp), HistoGraph achieves the top ROC-AUC on 3 of 4: molbbbp 72.02%, moltox21 77.49%, toxcast 66.35%. Margins over the second-best are +2.29% on molbbbp versus DKEPool, +0.91% on toxcast versus GMT, and +0.19% on moltox21 versus GMT. On molhiv, DKEPool leads with 78.65%, while HistoGraph is competitive at 77.81%, ranking in the top three, indicating strong generalization across molecular property prediction.

Node classification. Table 1 shows that HistoGraph mitigates over-smoothing: standard GCN accuracy degrades with depth, whereas HistoGraph maintains stable, competitive performance up to 64 layers. This improves feature propagation while preserving discriminative power, particularly on heterophilic graphs. Additional node-classification results for heterophilic and homophilic datasets appear in Table 4 in Appendix 12.

Post-Processing of Trained GNNs with HistoGraph

We evaluate HistoGraph as a lightweight post-processing strategy on four TU graph-classification datasets: imdb-b, imdb-m, proteins, and ptc. For each dataset, we train GINs with 5, 16, 32, and 64 layers using standard architectures and mean pooling. After convergence, we save per-fold checkpoints and apply HistoGraph in three modes: (i) auxiliary head on a frozen backbone (HistoGraph(FT)), (ii) full joint fine-tuning (HistoGraph(Full FT)), and (iii) end-to-end training from scratch for comparison. Complete depth-wise results appear in Table 10 in Appendix 12.

Table [tab:post-process-best-depth] summarizes the graph-classification accuracy (%) across GIN depths for each dataset and method. HistoGraph used as a frozen auxiliary head (FT) consistently improves performance vs. MeanPool, often matching or surpassing full fine-tuning (Full FT) and end-to-end training. For example, on imdb-m, FT raises accuracy from 54.7% (MeanPool) to 67.3%; on imdb-b, both FT and Full FT reach 94.0%, far above the baseline (76.0%) and end-to-end (87.2%). On proteins, all HistoGraph variants achieve near-optimal performance, demonstrating effectiveness across datasets of varying size and characteristics. On ptc, Full FT attains the best score (97.1%), showing joint fine-tuning can further enhance results. Overall, HistoGraph offers a flexible, effective post-processing strategy that consistently boosts GNN performance.

Runtime Analysis. We measure average training time per epoch for GCN backbones with 3 and 32 layers on molhiv and toxcast, comparing MeanPool, End-to-End, and FT. As shown in Fig. 8, End-to-End is costlier than MeanPool (e.g., 60.34s vs. 41.27s for 32 layers on molhiv) yet remains scalable. FT, which fine-tunes only the head on a pretrained MeanPool model, cuts overhead: training time is slightly higher for 3 layers but significantly lower for 32 layers on both datasets. While achieving results comparable to End-to-End (Table 10), FT offers an efficient way to boost existing models. Finally, HistoGraph is significantly faster than GMT  in almost all cases, with larger speedups for deeper networks.

Ablation Study

Setup. We assess component contributions on the TUDatasets proteins dataset by removing or modifying parts and measuring classification accuracy (Table [tab:ablation]). We test three variants: (i) removing division-by-sum normalization, (ii) disabling layer-wise attention that models inter-layer dependencies, and (iii) disabling node-wise attention that captures cross-node dependencies.

Results and discussion. On proteins, HistoGraph attains 97.80% accuracy with a 0.40 standard deviation. Every ablation reduces accuracy; removing division-by-sum normalization performs worst at 74.45% $`\pm`$ 6.28, indicating each component is necessary. Removing layer-wise normalization allows attention weights to grow unbounded, destabilizing training and overshadowing early discriminative layers. Our signed normalization balances layer contributions and enables additive and subtractive filtering (Section 4), preserving discriminative information and stability. Against alternative aggregation strategies (mean aggregation and randomized attention), HistoGraph consistently outperforms them by a significant margin (Table 9, Appendix 12). Overall, normalization, layer-wise attention, and node-wise attention are critical for capturing complex dependencies and realizing the full performance of HistoGraph.

Conclusion

We introduced HistoGraph, a two-stage attention-based pooling layer that learns from historical activations to produce stronger graph-level representations. The design is simple and principled: layer-wise attention captures the evolution of each node’s trajectory across depths, node-wise self-attention models spatial interactions at readout, and signed layer-wise normalization balances contributions across layers to preserve discriminative signals and stabilize training. This combination mitigates over-smoothing and supports deeper GNNs while keeping computation and memory overhead modest. Across TU and OGB graph-level benchmarks, and in node-classification settings, HistoGraph consistently improves over strong pooling baselines and matches or surpasses leading methods on multiple datasets. Moreover, used as a lightweight post-processing head on frozen backbones, HistoGraph delivers additional gains without retraining the encoder. Taken together, the results establish intermediate activations as a valuable signal for readout and position HistoGraph as a practical, general drop-in pooling layer for modern GNNs.

Reproducibility Statement

To ensure reproducibility, we provide all code, model architectures, training scripts, and hyperparameter settings in a public repository (available upon acceptance). Dataset preprocessing, splits, and downsampling are detailed in Appendix 7. Hyperparameter configurations, including batch sizes, learning rates, hidden dimensions, and model depths, are documented in Appendix 9.1. Experiments were conducted using PyTorch and PyTorch Geometric on NVIDIA L40, A100, and GeForce RTX 4090 GPUs, with Weights and Biases for logging and model selection. All random seeds and training protocols are specified to facilitate replication.

Ethics Statement

Our work involves minimal ethical concerns. We use publicly available datasets (TU, OGB) that are widely adopted in graph learning research, adhering to their licensing terms. No private or sensitive data is introduced. Our method is primarily methodological, but we encourage responsible use to avoid potential misuse in applications that could impact privacy or enable harm. We acknowledge the environmental impact of large-scale training and note that HistoGraph’s computational efficiency may reduce energy costs compared to retraining full models.

Usage of Large Language Models in This Work

Large language models were used solely for minor text editing suggestions to improve clarity and grammar. All research concepts, code development, experimental design, and original writing were performed by the authors.

Dataset Statistics

Tables 2 and 3 summarize the statistics of the datasets used in our experiments. Table 2 covers molecular property prediction datasets from the Open Graph Benchmark (OGB), including molhiv, molbbbp, moltox21, and toxcast, reporting the number of graphs, number of prediction classes, and average number of nodes per graph. Table 3 presents the statistics of graph classification datasets from the TU benchmark suite, including social network datasets (imdb-b, imdb-m) and bioinformatics datasets (mutag, ptc, proteins, rdt-b, nci1). These datasets vary widely in graph sizes and label space, providing a comprehensive evaluation setting across small, medium, and large graphs with diverse class distributions.

Implementation Details of HistoGraph

Algorithm [alg:method] outlines the forward pass of HistoGraph. The input $`\mathbf{X} \in \mathbb{R}^{N \times L \times D_\text{in}}`$ consists of node embeddings across $`L`$ GNN layers, for $`N`$ nodes per graph (referred to as historical graph activations). We first project the input to a common hidden dimension $`D`$ using a shared linear transformation. Sinusoidal positional encodings are added to encode the layer index. The final-layer embeddings serve as the query in an attention mechanism, while all intermediate layers act as key and value inputs. Attention scores are computed, averaged across nodes, and normalized over layers to yield a weighted aggregation of layer-wise features. A multi-head self-attention (MHSA) block is then applied over the aggregated node representations to capture spatial dependencies. Finally, a global average pooling operation over the node dimension produces the final graph-level representation $`\mathbf{Y} \in \mathbb{R}^D`$.

To stabilize training, we combined the output of HistoGraph with a simple mean pooling baseline using a learnable weighting factor $`\alpha \in [0, 1]`$. Specifically, the final graph representation was computed as a convex combination of the output of our method and the mean of the final-layer node embeddings: $`\mathbf{Y}_{\text{final}} = \alpha \cdot \mathbf{Y}_{\text{\textsc{HistoGraph}}} + (1 - \alpha) \cdot \mathbf{Y}_{\text{mean}}`$. We experimented both with fixed and learnable values of $`\alpha`$, and found that incorporating the mean-pooling signal helps guide the optimization in early training stages.

Experimental Details

We implemented our method using PyTorch  (offered under BSD-3 Clause license) and the PyTorch Geometric library  (offered under MIT license). All experiments were run on NVIDIA L40, NVIDIA A100 and GeForce RTX 4090 GPUs. For logging, hyperparameter tuning, and model selection, we used the Weights and Biases (W&B) framework .

In the subsection below, we provide details on the hyperparameter configurations used across our experiments.

Hyperparameters

The hyperparameters in our method include the batch size $`B`$, hidden dimension $`D`$, learning rate $`l`$, and weight decay $`\gamma`$. We also tune architectural and attention-specific components such as the number of attention heads $`H`$, use of fully connected layers, inclusion of zero attention token, use of layer normalization, and skip connections. Attention dropout rates are controlled via the multi-head attention dropout $`p_{\text{mha}}`$ and attention mask dropout $`p_{\text{mask}}`$. We further include the use of a learning rate schedule as a hyperparameter. Additionally, we consider different formulations for the attention coefficient parameterization $`\alpha_{\text{type}}`$, including learnable, fixed, and gradient-constrained variants. Hyperparameters were selected via a combination of grid search and Bayesian optimization, using validation performance as the selection criterion. For baseline models, we consider the search space of their relevant hyperparameters.

Additional Properties of HistoGraph

Contribution of Node-wise Attention for Graph-Level Prediction. Let $`\mathbf{H} = [\mathbf{h}_1, \dots, \mathbf{h}_N] \in \mathbb{R}^{N \times D}`$ be the cross-layer-pooled node embeddings. Suppose the downstream task requires a function $`f: \mathbb{R}^{N \times D} \to \mathbb{R}^K`$ that is permutation-invariant but non-uniform (e.g., depends on inter-node interactions). Then, standard mean pooling cannot approximate $`f`$ unless it includes additional inter-node operations like node-wise attention.

As a concrete example where the node-wise attention is beneficial in our HistoGraph, let us consider a graph $`G = (V, E)`$ composed of two subgraphs connected by a narrow bridge. Let $`G_L = (V_L, E_L)`$ be a large graph $`G(n, p)`$ with $`n \gg 1`$, and let $`G_R = (V_R, E_R)`$ be a singleton graph containing a single node $`v_R`$. The resulting structure is illustrated in Figure 9.

Suppose the graph-level classification task depends solely on the features of the singleton node $`v_R`$ (e.g., label is determined by a property encoded in $`v_R`$). In this setting, a naive mean pooling aggregates all node embeddings uniformly. As $`n`$ increases, the contribution of $`v_R`$ to the pooled representation becomes increasingly marginal, leading to its signal being dominated by the embeddings from the much larger subgraph $`G_L`$. This becomes especially problematic when there is a distribution shift at test time, e.g., $`G_L`$ becomes larger or denser, which further suppresses the contribution of $`v_R`$.

In contrast, a node-wise attention mechanism can learn to attend selectively to $`v_R`$, regardless of the size of $`G_L`$, making it robust to distributional changes. This demonstrates the contribution of node-wise attention in capturing non-uniform importance of nodes in our HistoGraph.

/>

/>

Proofs

HistoGraph  mitigates oversmoothing

Proof.

Consider two nodes $`u`$ and $`v`$. Under the definition of HistoGraph’s final embedding:

\begin{equation}
h_u - h_v = \sum_{l=0}^{L-1} \alpha_l \left( \mathbf{x}_u^{(l)} - \mathbf{x}_v^{(l)} \right).
\end{equation}

We split the sum into layers before and after $`L_0`$:

\begin{align}
h_u - h_v 
&= \sum_{l=0}^{L_0} \alpha_l \left( \mathbf{x}_u^{(l)} - \mathbf{x}_v^{(l)} \right) 
+ \sum_{l=L_0+1}^{L-1} \alpha_l \left( \mathbf{x}_u^{(l)} - \mathbf{x}_v^{(l)} \right).
\end{align}

By over-smoothing (Eq. [eq:oversmooth-def]), for all $`l > L_0`$:

\begin{equation}
\| \mathbf{x}_u^{(l)} - \mathbf{x}_v^{(l)} \| \approx 0,
\end{equation}

and hence the second sum is negligible. Therefore,

\begin{equation}
h_u - h_v \approx \sum_{l=0}^{L_0} \alpha_l \left( \mathbf{x}_u^{(l)} - \mathbf{x}_v^{(l)} \right).
\end{equation}

Because initial node representations differ (a standard assumption for distinct nodes), there exists at least one layer $`l' \leq L_0`$ for which

\begin{equation}
\| \mathbf{x}_u^{(l')} - \mathbf{x}_v^{(l')} \| \neq 0.
\end{equation}

Given that HistoGraph employs learned dynamic attention, suppose $`\alpha_{l'} \neq 0`$. Consequently:

\begin{align}
\| h_u - h_v \| 
&\approx \left\| \alpha_{l'} \left( \mathbf{x}_u^{(l')} - \mathbf{x}_v^{(l')} \right) 
+ \sum_{l \neq l'} \alpha_l \left( \mathbf{x}_u^{(l)} - \mathbf{x}_v^{(l)} \right) \right\| \\
&> 0.
\end{align}

This directly contradicts the assumption that node embeddings become indistinguishable in the pooled representation. Thus, HistoGraph mitigates over-smoothing by explicitly retaining discriminative early-layer representations.

Additional Experiments

Table [tab:full_turesults] and Table [tab:full_ogb_results] show the results of different methods on two different settings of multiple graph pooling methods. Table 4 reports node classification accuracy on both heterophilic and homophilic datasets. We observe that our method (HistoGraph+ GCN) consistently outperforms standard GCN and JKNet across all datasets. The improvements are particularly pronounced on heterophilic graphs such as Actor, Squirrel, and Chameleon, where our method achieves gains of up to 12.3% over GCN. On homophilic datasets like Cora, Citeseer, and Pubmed, we also observe consistent, albeit smaller, improvements.

To evaluate the ability of HistoGraph to mitigate oversmoothing, we measure the feature distance across layers for a standard GCN, both with and without HistoGraph. The results, presented in Table 5, show that incorporating HistoGraph consistently leads to higher feature distances across all layers, with the most pronounced improvement observed in the pre-classifier layer as expected due to the oversmoothing.

The results in Tables 6–8 further demonstrate the versatility and effectiveness of HistoGraph across different tasks and architectures. On the OGBL-COLLAB link prediction benchmark (Table 6), incorporating HistoGraph as a readout function leads to consistent improvements over a standard GCN baseline. Similarly, in molecular property prediction tasks with GraphGPS backbones (Table 7), HistoGraph achieves substantial performance gains across multiple datasets, highlighting its ability to preserve and leverage historical information across layers. Finally, the ablation on the number of historical layers (Table 8) shows that incorporating deeper historical context enhances predictive performance, with the best results obtained when more layers are retained. These findings underscore the robustness of HistoGraph as a drop-in replacement for readout functions across diverse settings. Finally, we present an additional ablation in Table 9, which examines the effect of different aggregation strategies across all layers—mean aggregation, randomized attention, and HistoGraph. Across datasets, HistoGraph consistently achieves superior performance.

Scalable Post-Processing with HistoGraph

Table 10 indicates that all HistoGraph variants consistently outperform the MeanPool baseline for every depth and dataset. In particular, FT, often matches or even exceeds the accuracy of full end‐to‐end tuning despite having far fewer trainable parameters. For example, at 5 layers it boosts imdb-m from 54.0 % to 67.3 %, imdb-b accuracy from 76.0 % to 94.0 %, proteins from 75.0 % to 97.3 %, and ptc from 77.1 % to 85.7 %. As model depth grows, FT remains highly competitive: at 16 layers it achieves 64.7%.

We would like to note that while HistoGraph mitigates over-smoothing by dynamically leveraging early-layer discriminative features, at extreme depths (e.g., 64 layers) we face known optimization challenges in GNNs . Nonetheless, HistoGraph  consistently outperforms baseline pooling methods, as shown in our depth-varying experiments on Cora, Citeseer, and Pubmed in Table 1, demonstrating robustness to model depth.

These findings demonstrate that caching intermediate representations and training a small auxiliary head enables scalable, modular adaptation of GNNs, obtaining strong performance across depths and domains without incurring the computational costs of full model training.