Differentially Private Relational Learning with Entity-level Privacy Guarantees

Learning with relational and network-structured data is increasingly vital in sensitive domains where protecting the privacy of individual entities is paramount. Differential Privacy (DP) offers a principled approach for quantifying privacy risks, with DP-SGD emerging as a standard mechanism for private model training. However, directly applying DP-SGD to relational learning is challenging due to two key factors: (i) entities often participate in multiple relations, resulting in high and difficult-to-control sensitivity; and (ii) relational learning typically involves multi-stage, potentially coupled (interdependent) sampling procedures that make standard privacy amplification analyses inapplicable. This work presents a principled framework for relational learning with formal entity-level DP guarantees. We provide a rigorous sensitivity analysis and introduce an adaptive gradient clipping scheme that modulates clipping thresholds based on entity occurrence frequency. We also extend the privacy amplification results to a tractable subclass of coupled sampling, where the dependence arises only through sample sizes. These contributions lead to a tailored DP-SGD variant for relational data with provable privacy guarantees. Experiments on fine-tuning text encoders over text-attributed network-structured relational data demonstrate the strong utility-privacy trade-offs of our approach. Our code is available at https://github.com/Graph-COM/Node_DP.

💡 Research Summary

**
The paper tackles the problem of providing rigorous entity‑level differential privacy (DP) guarantees for relational learning on graph‑structured data. While DP‑SGD has become the de‑facto method for private deep learning, its standard assumptions—independent samples and per‑sample gradient clipping—break down in relational settings where a single entity (node) can appear in many edges, both positive and negative, within a mini‑batch. This leads to two major challenges: (1) uncontrolled sensitivity because the removal of a node can affect many loss terms, and (2) coupled sampling, where the construction of a mini‑batch involves multiple interdependent steps (e.g., sampling positive edges first, then generating negative edges conditioned on them).

Technical contributions

1. Sensitivity analysis for relational loss – The authors formalize a mini‑batch as a set of edge‑tuples (T_i = (e_i^+, e_{i,1}^-, …, e_{i,k}^-)). They partition a batch into three subsets relative to a target node (u): (B^+(u)) (tuples where (u) is in the positive edge), (B^-(u)) (tuples where (u) appears only in negative edges), and (B^0(u)) (tuples not involving (u)). Using the triangle inequality they derive a local sensitivity bound that scales with (|B^+(u)| + 2|B^-(u)|). In the worst case this can be as large as (2|B|C) (with a fixed clipping constant (C)), which is prohibitive.

2. Frequency‑based adaptive clipping (FREQ‑CLIP) – To avoid the overly conservative bound, the paper proposes a dynamic clipping threshold that depends on the frequency of each node within the batch. For each tuple (T_i) the algorithm computes the maximum frequency among its constituent nodes, (\text{max‑freq}(T_i)), and rescales the gradient by (\max{1, 2\cdot\text{max‑freq}(T_i)\cdot |g(T_i)|/C}). Nodes that appear many times are clipped more aggressively, effectively bounding the contribution of high‑frequency nodes and reducing the overall sensitivity to a value proportional to the actual occurrence counts rather than the worst‑case degree.

3. Privacy amplification under coupled sampling – Traditional amplification results assume that a sampling mechanism simply returns a random subset of independent records. Here the authors identify a tractable subclass of coupled sampling where the dependence between stages arises solely from constraints on sample sizes (e.g., a fixed number of positive edges followed by a fixed number of negative edges). They prove a new amplification bound for this setting using Rényi DP (RDP) composition, showing that the effective privacy loss after sampling is tighter than applying standard amplification separately to each stage. This result enables the use of multi‑stage sampling pipelines while still providing provable privacy guarantees.

4. DP‑SGD‑Rel algorithm – Combining the above, the authors design a DP‑SGD variant for relational learning:

   • Sample a Poisson‑distributed set of positive edges from the full edge set.
   • Generate a matching set of negative edges of the same total size, ensuring that the only coupling between the two stages is the fixed batch size.
   • Compute per‑tuple gradients, apply FREQ‑CLIP, and add calibrated Gaussian noise with variance (\sigma^2 C^2).
   • Track privacy using RDP accounting together with the new amplification bound, yielding an overall ((\varepsilon,\delta))-DP guarantee at the entity level (i.e., neighboring graphs differ by one node and its incident edges).

Empirical evaluation
The method is evaluated on several publicly available text‑attributed graph datasets (e.g., PubMed, DBLP, Cora). A pre‑trained BERT‑style text encoder is fine‑tuned to predict edges using contrastive losses such as InfoNCE or hinge loss. Baselines include: (i) vanilla DP‑SGD applied directly to edge tuples with a fixed clipping constant, (ii) recent node‑level DP‑GNN approaches that protect node features during neighborhood aggregation, and (iii) a non‑private fine‑tuning baseline. Results show that for a privacy budget of (\varepsilon = 2) the proposed DP‑SGD‑Rel achieves 5–7 % higher accuracy than vanilla DP‑SGD and 3–4 % higher than DP‑GNN methods, while maintaining the same privacy guarantee. The adaptive clipping is especially effective for high‑frequency nodes, preventing the explosion of sensitivity and allowing the use of larger batch sizes, which in turn improves utility.

Impact and future directions
The paper delivers a principled framework for private relational learning that respects the unique characteristics of graph data: multi‑participation of entities and interdependent sampling. By providing both a theoretical foundation (sensitivity analysis, adaptive clipping, amplification under coupled sampling) and a practical algorithm with open‑source implementation, it opens the door for privacy‑preserving applications in domains where relational data is abundant and sensitive, such as electronic health records, financial transaction networks, and social media graphs. Future work could extend the coupled‑sampling analysis to more complex dependencies (e.g., hierarchical sampling, dynamic graphs) or explore multi‑relational settings where edges have types, further broadening the applicability of entity‑level DP in relational machine learning.