Model Editing with Graph-Based External Memory

Large language models (LLMs) have revolutionized natural language processing, yet their practical utility is often limited by persistent issues of hallucinations and outdated parametric knowledge. Although post-training model editing offers a pathway for dynamic updates, existing methods frequently suffer from overfitting and catastrophic forgetting. To tackle these challenges, we propose a novel framework that leverages hyperbolic geometry and graph neural networks for precise and stable model edits. We introduce HYPE (HYperbolic Parameter Editing), which comprises three key components: (i) Hyperbolic Graph Construction, which uses Poincaré embeddings to represent knowledge triples in hyperbolic space, preserving hierarchical relationships and preventing unintended side effects by ensuring that edits to parent concepts do not inadvertently affect child concepts; (ii) Möbius-Transformed Updates, which apply hyperbolic addition to propagate edits while maintaining structural consistency within the hyperbolic manifold, unlike conventional Euclidean updates that distort relational distances; and (iii) Dual Stabilization, which combines gradient masking and periodic GNN parameter resetting to prevent catastrophic forgetting by focusing updates on critical parameters and preserving long-term knowledge. Experiments on CounterFact, CounterFact+, and MQuAKE with GPT-J and GPT2-XL demonstrate that HYPE significantly enhances edit stability, factual accuracy, and multi-hop reasoning.

💡 Research Summary

The paper addresses the persistent problem of factual errors and outdated knowledge in large language models (LLMs) by proposing a novel post‑training model‑editing framework called HYPE (Hyperbolic Parameter Editing). Existing editing techniques such as ROME, MEMIT, and MEND operate in Euclidean space and often suffer from over‑fitting, catastrophic forgetting, and unintended side effects when hierarchical relationships are involved. HYPE overcomes these limitations by embedding a knowledge graph into a hyperbolic manifold (the Poincaré ball) and performing parameter updates with Möbius addition, which respects the curvature of the space.

The framework consists of three core components. First, Hyperbolic Graph Construction builds a directed graph from Wikipedia triples, obtains initial Euclidean embeddings via a graph neural network (GNN), and projects them into hyperbolic space using the exponential map with a learnable curvature parameter c (set to 1.0 in experiments). This projection preserves hierarchical distances: child nodes are placed exponentially farther from their parents, enabling edits that affect only the intended level of the hierarchy. A persistence filter further prunes low‑importance relations, ensuring that only structurally significant edges influence subsequent updates.

Second, Möbius‑Transformed Updates replace the standard Euclidean weight addition w′ = w + Δ with Möbius addition w ⊕c Δ. The base update Δ is generated by the rank‑1 method (outer product of left and right vectors u and v) and scaled by a residual factor γ. To focus updates on salient parameters, the method computes the average gradient magnitude for each output dimension, applies a sigmoid mask, and multiplies the mask element‑wise with Δ. The resulting masked Δ is then combined with the current weight via Möbius addition, guaranteeing that the new weight remains on the hyperbolic manifold and that hierarchical relationships are not distorted.

Third, Dual Stabilization introduces two safeguards. After each Möbius update, a projection operator forces the weight back into the valid region of the Poincaré ball (‖w‖ ≤ 1/√c), preventing numerical drift. Additionally, the GNN that produces the update directions is reset to its initial parameters after every editing cycle. This reset stops the GNN from over‑fitting to transient patterns of a single edit, thereby mitigating catastrophic forgetting and preserving long‑term knowledge.

The authors evaluate HYPE on three benchmark datasets: CounterFact (single‑fact edits), CounterFact+ (paraphrase robustness), and MQuAKE (multi‑hop reasoning). Experiments are conducted on two LLMs, GPT‑J (6 B parameters) and GPT2‑XL (1.5 B parameters), and compared against seven strong baselines: Zero‑shot, Fine‑Tuning, MEND, ROME, MEMIT, PMET, and RAE. Metrics include Edit Efficiency (accuracy after edit), Generalization (performance on paraphrased queries and reasoning tasks), and Edit Quality Score (EDS).

Results show that HYPE consistently outperforms all baselines. It achieves an average increase of +9.12 points in EDS and +3.59 points in edit efficiency. Notably, hierarchical edits (e.g., modifying “mammal” without altering its child “whale”) exhibit minimal side effects, confirming the benefit of hyperbolic embeddings. Gradient masking and GNN resetting dramatically reduce catastrophic forgetting, preserving previously learned facts even after multiple successive edits. Computational overhead remains modest because Möbius addition is a closed‑form operation and the GNN is lightweight; thus HYPE is suitable for real‑time knowledge updates.

The paper also discusses limitations and future directions. The curvature parameter c is fixed during training; learning a dynamic curvature could further adapt the embedding space to diverse knowledge structures. Current experiments rely on a static Wikipedia graph; integrating streaming knowledge sources would require incremental graph updates and continual hyperbolic re‑embedding. The GNN reset strategy, while effective for isolated edits, may discard useful adaptation when edits are issued in rapid succession, suggesting a need for more sophisticated memory mechanisms.

In conclusion, HYPE demonstrates that hyperbolic geometry provides a principled way to encode hierarchical knowledge and that Möbius‑based parameter updates can edit LLMs precisely without compromising existing knowledge. The framework achieves superior factual accuracy, edit stability, and multi‑hop reasoning capability across multiple models and datasets, opening a promising path toward scalable, real‑time model maintenance. Future work may explore dynamic curvature learning, integration with external memory stores, and extensions to multimodal models.