<SOG_k>: One LLM Token for Explicit Graph Structural Understanding

Large language models show great potential in unstructured data understanding, but still face significant challenges with graphs due to their structural hallucination. Existing approaches mainly either verbalize graphs into natural language, which leads to excessive token consumption and scattered attention, or transform graphs into trainable continuous embeddings (i.e., soft prompt), but exhibit severe misalignment with original text tokens. To solve this problem, we propose to incorporate one special token <SOG_k> to fully represent the Structure Of Graph within a unified token space, facilitating explicit topology input and structural information sharing. Specifically, we propose a topology-aware structural tokenizer that maps each graph topology into a highly selective single token. Afterwards, we construct a set of hybrid structure Question-Answering corpora to align new structural tokens with existing text tokens. With this approach, <SOG_k> empowers LLMs to understand, generate, and reason in a concise and accurate manner. Extensive experiments on five graph-level benchmarks demonstrate the superiority of our method, achieving a performance improvement of 9.9% to 41.4% compared to the baselines while exhibiting interpretability and consistency. Furthermore, our method provides a flexible extension to node-level tasks, enabling both global and local structural understanding. The codebase is publicly available at https://github.com/Jingyao-Wu/SOG.

💡 Research Summary

The paper tackles a fundamental bottleneck in applying large language models (LLMs) to graph‑structured data: how to feed non‑Euclidean topology into a model that expects a linear token stream without incurring massive token overhead or suffering from modality mismatch. Existing solutions fall into two camps. Graph‑to‑Text methods verbalize edges, nodes, and attributes, which quickly exhaust the token budget and are highly sensitive to the ordering of descriptions, leading to “structural hallucination.” Graph‑to‑Embedding approaches compress the graph into continuous vectors (soft prompts) and project them into the LLM’s embedding space, but the projected vectors remain poorly aligned with the discrete token vocabulary, resulting in weak cross‑modal compatibility.

The authors propose a novel “Graph‑to‑Token” paradigm centered on a single special token <SOG_k> that encapsulates the entire graph topology within the same vocabulary as ordinary text tokens. Their solution consists of two stages:

1. Topology‑Aware Graph Structural Tokenizer

   • The raw graph G(V,E) is stripped of all non‑structural attributes. Each node receives a hierarchical positional label (e.g., “first‑hop neighbor #3”) based on an anchor node selected by a simple importance metric such as degree. A virtual global node is added to aggregate a whole‑graph representation.
   • These textual labels are embedded with a standard text encoder f_T, producing node‑level vectors X_s. A Graph Neural Network f_G then performs message passing, yielding continuous node embeddings H_s that capture multi‑hop topology.
   • To bridge the continuous space and the discrete token space, a learnable codebook C = {c₁,…,c_K} is introduced (similar to VQ‑VAE). Each node embedding h_i^s is quantized to its nearest codebook entry c_k via Euclidean distance. The index of the codebook entry assigned to the global node becomes the identifier of the structural token <SOG_k>.
   • Training uses three losses: (i) reconstruction loss of the adjacency matrix via a lightweight decoder f_q, (ii) an update loss encouraging the selected codebook entry to move toward the continuous embedding, and (iii) a commitment loss preventing codebook collapse. The overall loss L = L_recon + L_update + β L_commit aligns the discrete token with the underlying topology.

2. Token Alignment via Hybrid Structure QA

   • After obtaining <SOG_k>, the authors construct three families of question‑answer pairs that involve only structural information:
     a) k‑Nearest Token Neighbor Matching – asks for the top‑k nearest structural tokens to a given <SOG_i>, enforcing local similarity in the embedding space.
     b) True/False Structure Similarity Judgment – presents two tokens and asks whether they represent similar graphs, using a cosine‑similarity threshold on the underlying continuous embeddings to generate “similar” or “dissimilar” labels. This acts as contrastive supervision.
     c) Description‑Token Pair Matching – provides a textual description of a small subgraph and asks for the corresponding structural token, directly linking text and token representations.
   • The LLM is fine‑tuned on these QA pairs while freezing all original text token embeddings; only the newly added structural token embeddings are updated. Additionally, a lightweight LoRA adapter injects structural knowledge without disturbing the pretrained weights. This stage aligns <SOG_k> with the LLM’s existing vocabulary, allowing the model to treat the token as a first‑class citizen during downstream inference.

Downstream Evaluation
The method is evaluated on five graph‑level benchmarks (including ZINC, ogbg‑molpcba, PCQM4M, and two others). Across all datasets, the proposed approach outperforms strong baselines—both Graph‑to‑Text (e.g., InstructGraph, Talk Like a Graph) and Graph‑to‑Embedding (e.g., GraphGPT, LLaGA)—by 9.9 % to 41.4 % absolute improvement in standard metrics (e.g., ROC‑AUC, MAE). Notably, the entire graph is represented by a single token, drastically reducing input length and computational cost. The authors also demonstrate extension to node‑level classification tasks, where a global token captures overall structure while optional local tokens encode neighborhood information, achieving competitive results.

Strengths

• Token Efficiency: One token replaces potentially thousands of textual edge descriptors, preserving the LLM’s context window.
• Alignment: The hybrid QA fine‑tuning directly aligns structural tokens with the LLM’s embedding space, mitigating the modality gap that plagues soft‑prompt methods.
• Scalability: Because the tokenizer outputs a discrete index, inference adds negligible overhead; the method can be plugged into any pretrained LLM without architectural changes.
• Interpretability: The codebook entries can be inspected to understand which structural motifs they correspond to, offering a degree of transparency absent in continuous embeddings.

Limitations and Open Questions

• Codebook Size vs. Expressivity: The hyperparameter K determines how finely the space of possible graph topologies is discretized. Small K may collapse distinct graphs into the same token, while large K increases the vocabulary and may require more data to learn reliable mappings.
• Large‑Scale Graphs: For graphs with tens of thousands of nodes, compressing the entire topology into a single token may discard crucial local patterns. A hierarchical or multi‑token extension could be necessary.
• Dependency on GNN Pretraining: The tokenizer relies on a GNN encoder trained jointly with the codebook. If the GNN is poorly suited to a particular domain (e.g., molecular graphs vs. social networks), the resulting token may be suboptimal.
• Text‑Structure Interaction: The QA set focuses on pure structural alignment; it does not explicitly teach the model how structural tokens interact with textual node/edge attributes, which could limit performance on tasks where attribute‑structure synergy is critical.
• Generalization Across Domains: While the authors claim strong transferability because the QA data is structure‑only, empirical validation on completely unseen graph domains (e.g., knowledge graphs with rich predicates) would strengthen this claim.

Future Directions

• Hierarchical Tokenization: Introduce a sequence of <SOG_k> tokens representing sub‑graphs or hierarchical clusters, enabling scalable representation of massive graphs.
• Dynamic Codebooks: Allow the codebook to grow or adapt during downstream fine‑tuning, preserving flexibility while maintaining alignment.
• Joint Text‑Structure Pretraining: Create mixed QA pairs that involve both attribute text and structural tokens, encouraging the model to learn cross‑modal reasoning directly.
• Benchmark Expansion: Test on knowledge‑graph completion, recommendation, and program analysis tasks to assess the universality of the approach.
• Integration with Retrieval: Combine <SOG_k> with external graph retrieval mechanisms, where the token can act as a compact identifier for retrieved subgraphs.

In summary, the paper presents a compelling and technically sound framework that redefines how LLMs can ingest graph data. By compressing an entire graph topology into a single, vocabulary‑compatible token and aligning it through carefully crafted QA supervision, the authors achieve substantial performance gains while dramatically reducing token consumption. The work opens a promising research avenue toward unified language‑graph models, though further exploration is needed to address scalability, codebook granularity, and richer text‑structure interactions.