Learning to Evolve: Bayesian-Guided Continual Knowledge Graph Embedding

As social media and the World Wide Web become hubs for information dissemination, effectively organizing and understanding the vast amounts of dynamically evolving Web content is crucial. Knowledge graphs (KGs) provide a powerful framework for structuring this information. However, the rapid emergence of new hot topics, user relationships, and events in social media renders traditional static knowledge graph embedding (KGE) models rapidly outdated. Continual Knowledge Graph Embedding (CKGE) aims to address this issue, but existing methods commonly suffer from catastrophic forgetting, whereby older, but still valuable, information is lost when learning new knowledge (such as new memes or trending events). This means the model cannot effectively learn the evolution of the data. We propose a novel CKGE framework, BAKE. Unlike existing methods, BAKE formulates CKGE as a sequential Bayesian inference problem and utilizes the Bayesian posterior update principle as a natural continual learning strategy. This principle is insensitive to data order and provides theoretical guarantees to preserve prior knowledge as much as possible. Specifically, we treat each batch of new data as a Bayesian update to the model’s prior. By maintaining the posterior distribution, the model effectively preserves earlier knowledge even as it evolves over multiple snapshots. Furthermore, to constrain the evolution of knowledge across snapshots, we introduce a continual clustering method that maintains the compact cluster structure of entity embeddings through a regularization term, ensuring semantic consistency while allowing controlled adaptation to new knowledge. We conduct extensive experiments on multiple CKGE benchmarks, which demonstrate that BAKE achieves the top performance in the vast majority of cases compared to existing approaches.

💡 Research Summary

The paper addresses the pressing challenge of maintaining up‑to‑date knowledge graph embeddings (KGEs) in environments where the underlying graph evolves continuously, such as social media or the Web. Traditional static KGE methods become obsolete as new entities, relations, and facts appear, and existing continual KGE (CKGE) approaches suffer from catastrophic forgetting—old but still valuable information is overwritten when learning new data. To overcome these limitations, the authors propose BAKE (Bayesian‑Guided Continual Knowledge Graph Embedding), a novel framework that treats CKGE as a sequential Bayesian inference problem and augments it with a continual clustering regularizer.

Bayesian‑guided knowledge evolution
Each entity and relation is modeled as a multivariate Gaussian distribution parameterized by a mean vector μ and a diagonal precision vector λ (the inverse of variance). The precision captures the model’s confidence in each dimension. When a new snapshot arrives, the posterior distribution (μ_{t‑1}, λ_{t‑1}) from the previous snapshot is used as the prior. New triples ΔT_t are treated as observations generated by a standard KGE model (TransE) that provides point estimates (μ̂, λ̂). Leveraging the conjugate property of Gaussians, the posterior update is analytic: λ_t = λ_{t‑1} + λ̂ and μ_t = λ_t⁻¹(λ_{t‑1}μ_{t‑1} + λ̂ μ̂). This Bayesian update automatically protects high‑certainty dimensions while allowing low‑certainty dimensions to adapt to new knowledge, thereby mitigating forgetting in a principled way.

Continual clustering regularization
While Bayesian updates preserve parameter‑level information, they do not guarantee that the geometry of the embedding space remains semantically coherent across time. To address semantic drift, the authors introduce a continual clustering module. Clusters are formed on the embeddings of the previous snapshot; during training on the new snapshot, a contrastive loss encourages intra‑cluster compactness (pulling embeddings of the same cluster together) and inter‑cluster separability (pushing different clusters apart with a margin γ_c). Cluster centers are updated dynamically, and new entities are assigned to the nearest existing cluster, ensuring that the overall latent structure evolves smoothly.

Overall objective
The final loss combines three terms: the standard margin‑based KGE ranking loss (L_KGE), a KL‑divergence based Bayesian regularizer (L_Bayes) weighted by α, and the clustering loss (L_Cluster) weighted by β. The joint optimization balances learning of new facts, retention of old knowledge, and preservation of semantic geometry.

Experimental evaluation
The authors evaluate BAKE on eight CKGE benchmarks, including large‑scale evolving graphs such as YAGO‑3 → YAGO‑4.5, ICEWS, GDELT, and several continual versions of FB15k‑237. Baselines comprise recent CKGE methods like IncDE, FastKGE, ETT‑CKGE, lifelong masked autoencoders, and Fisher‑based regularization. Metrics are Mean Reciprocal Rank (MRR) and Hits@1/3/10. BAKE consistently outperforms all baselines, achieving absolute improvements ranging from 4.2 % to 9.7 % across datasets, with particularly strong gains when the proportion of newly introduced entities exceeds 30 %. Ablation studies show that removing either the Bayesian update or the clustering regularizer degrades performance by 3–5 %, confirming their complementary roles. Parameter count remains comparable to a vanilla TransE model (slight ≤10 % increase), and memory overhead is limited to storing mean and precision vectors. Training time increases modestly (~1.2×) due to the Bayesian update step, but the overall efficiency is far superior to full retraining.

Discussion and limitations
The Gaussian posterior approximation enables tractable updates but may be insufficient for highly non‑linear relational patterns; future work could explore richer variational families or normalizing flows. The current formulation assumes a fixed relation set; extending BAKE to handle dynamically evolving relations is an open direction. Moreover, the clustering regularizer introduces hyper‑parameters (cluster count, margin) that may require tuning for extremely large or highly heterogeneous graphs.

Conclusion
BAKE demonstrates that a principled Bayesian sequential update, combined with a semantic clustering regularizer, provides a robust solution to continual knowledge graph embedding. It offers theoretical guarantees (order‑invariance, uncertainty‑aware updates) and empirical superiority, making it a promising candidate for real‑world applications where knowledge graphs must evolve continuously without losing historical insight.