GraphPFN: A Prior-Data Fitted Graph Foundation Model

Graph foundation models face several fundamental challenges including transferability across datasets and data scarcity, which calls into question the very feasibility of graph foundation models. However, despite similar challenges, the tabular domain has recently witnessed the emergence of the first successful foundation models such as TabPFNv2 and LimiX. Many of these models are based on the prior-data fitted networks (PFN) framework, in which models are pretrained on carefully designed synthetic datasets to make predictions in an in-context learning setting. Recently, G2T-FM has made the first step towards adopting PFNs for graphs, yet it is limited to hand-crafted features and was never pretrained on graph data. In this work, we make the next step by proposing GraphPFN, a PFN-based model designed and pretrained specifically for graph node-level tasks. Following the PFN framework, we first design a prior distribution of synthetic attributed graphs by using a novel combination of multi-level stochastic block models and a preferential attachment process for structure generation and graph-aware structured causal models for attribute generation. Then, we augment the tabular foundation model LimiX with attention-based graph neighborhood aggregation layers and train it on synthetic graphs sampled from our prior. On diverse real-world graph datasets with node-level tasks, GraphPFN shows strong in-context learning performance and achieves state-of-the-art results after finetuning, outperforming both G2T-FM and task-specific GNNs trained from scratch on most datasets. More broadly, GraphPFN shows the potential of PFN-based models for building graph foundation models.

💡 Research Summary

GraphPFN introduces a novel foundation model for graph‑structured data by extending the Prior‑Data Fitted Network (PFN) paradigm, which has proven successful in the tabular domain, to the graph domain. The authors first identify the core challenges of graph foundation models: extreme heterogeneity across datasets (different structures, feature spaces, and label types) and a scarcity of large, diverse graph corpora for pre‑training. To address these issues, they design a comprehensive synthetic graph prior that simultaneously captures realistic topology and attribute distributions.

For topology generation, they combine multiple stochastic block models (SBMs) to create community‑level structure and augment these with a preferential‑attachment process, thereby reproducing both modularity and scale‑free degree distributions observed in real networks. For node attributes, they adapt the structured causal models (SCMs) used in tabular PFNs, inserting message‑passing operations at randomly selected SCM nodes. This yields graph‑aware causal generation where a node’s features and label are conditionally dependent on its local neighborhood, enabling the creation of millions of diverse, realistic attributed graphs.

The model architecture builds on LimiX, a transformer‑style tabular foundation model that represents each sample as a grid of feature tokens. To inject graph information without breaking this tokenization, the authors add a graph‑adjacency‑masked attention adapter after every LimiX transformer block. This adapter performs a second round of sample‑level attention restricted to 1‑hop neighbors, effectively providing a GNN‑like message‑passing step within the transformer. The adapter is a sparse multi‑head attention module followed by a feed‑forward network, both wrapped with residual connections and layer normalization, mirroring the base block design. During pre‑training, all LimiX parameters are frozen, and only the adapters are updated, preserving the rich feature modeling already learned by LimiX.

Pre‑training follows the PFN protocol: 2.24 million synthetic datasets are sampled from the graph prior and processed in two stages. Stage 1 uses smaller graphs (1 k–2 k nodes) for 10 k optimizer steps; Stage 2 scales up to 8 k–10 k nodes for another 4 k steps. The loss combines the standard PFN supervised term (predicting query labels from context) with a masked graph modeling (MGM) loss that encourages the model to reconstruct masked node attributes and edges. Training runs for roughly seven days on eight NVIDIA A100 80 GB GPUs, with AdamW optimizer, cosine annealing schedule, and exponential moving average (EMA) for stability.

Evaluation covers twelve heterogeneous node‑level benchmarks spanning social networks, recommendation systems, transportation graphs, and text‑derived graphs. Two evaluation regimes are reported: (1) in‑context learning, where the model receives a mixed context‑query batch and must predict queries without weight updates, and (2) fine‑tuning, where the entire model (including adapters) is trained on the downstream task. In‑context results show GraphPFN matching or surpassing G2T‑FM and strong GNN baselines, demonstrating that the synthetic prior successfully transfers to real data. After fine‑tuning, GraphPFN achieves state‑of‑the‑art performance on the majority of datasets, outperforming both traditional GNNs (e.g., GCN, GAT, GraphSAGE with modern architectural tweaks) and recent graph foundation models.

Key contributions are: (i) the first publicly available PFN‑based graph foundation model specifically pre‑trained on synthetic graphs, (ii) a novel graph prior that efficiently generates realistic attributed graphs by merging SBMs, preferential attachment, and graph‑aware SCMs, and (iii) empirical evidence that pre‑training on such synthetic data yields strong transfer to diverse real‑world graph tasks, both in zero‑shot (in‑context) and fine‑tuned settings. The paper also discusses limitations—current focus on node‑level tasks, the need for larger‑scale graph‑level pre‑training, and potential extensions to subgraph or whole‑graph prediction, multimodal graph‑text/video models, and continual learning across graph domains. Overall, GraphPFN demonstrates that PFN‑style pre‑training is a promising pathway toward universal, data‑efficient graph foundation models.