Can LLM Safety Be Ensured by Constraining Parameter Regions?

Large language models (LLMs) are often assumed to contain ``safety regions’’ – parameter subsets whose modification directly influences safety behaviors. We conduct a systematic evaluation of four safety region identification methods spanning different parameter granularities, from individual weights to entire Transformer layers, across four families of backbone LLMs with varying sizes. Using ten safety identification datasets, we find that the identified safety regions exhibit only low to moderate overlap, as measured by IoU. The overlap drops significantly when the safety regions are further refined using utility datasets (\ie non-harmful queries). These results suggest that current techniques fail to reliably identify a stable, dataset-agnostic safety region.

💡 Research Summary

The paper investigates whether large language models (LLMs) contain well‑defined “safety regions” – subsets of parameters whose modification directly influences safety‑related behavior – and whether such regions can be reliably identified across different datasets. Four recent safety‑region identification methods are examined: SNIP & Wanda (parameter‑level importance scoring), SafeNeuron (neuron‑level scoring), SafeLayer (layer‑level selection based on cosine similarity and over‑rejection phenomena), and NLSR (LoRA‑weight‑level magnitude‑based selection). Each method assigns importance scores to components using harmful queries and selects the top‑percentile as the safety region.

The authors formulate two research questions: (RQ1) Does the overlap among safety regions identified from multiple identification datasets converge to a stable region as the number of datasets increases? (RQ2) How does the semantic distribution of the identification datasets (multi‑category vs. single‑category harm) affect the identified regions? To answer these, they construct ten multi‑category safety datasets and several single‑category datasets from PKU‑SafeRLHF‑QA and PKU‑SafeRLHF‑30K, covering diverse harm categories. A utility dataset (Alpaca‑Cleaned without safety‑related samples) is also used to derive utility regions, enabling measurement of “utility‑isolated” safety overlap.

Overlap is quantified with Intersection‑over‑Union (IoU) across the sets of component indices. Experiments span multiple LLM families (including Qwen, LLaMA, GPT‑Neo) and model sizes, following original hyper‑parameters and thresholds (e.g., q % for SNIP/Wanda, m = 1000/2000 for SafeNeuron, t = 20 % for NLSR).

Key findings: (1) SafeLayer could not be reproduced; the claimed middle‑layer safety zone does not manifest consistently. (2) For SNIP & Wanda, SafeNeuron, and NLSR, IoU among safety regions derived from different multi‑category datasets remains low (≈0.2–0.4) and declines further as more datasets are added. (3) When safety regions are refined by subtracting the utility region, IoU drops even more, indicating substantial entanglement between safety‑critical and utility‑critical parameters. (4) Model scale and architecture have only marginal impact on overlap; larger models show slight improvement but not enough to claim stable safety regions.

The authors conclude that current methods fail to satisfy the proposed “convergent identifiability” criterion: safety regions are highly fragmented and dataset‑dependent. Consequently, constraining identified regions during downstream fine‑tuning may not reliably preserve safety, as the regions lack stability and are interwoven with utility‑related components. The paper calls for more principled approaches that can disentangle safety from utility, possibly through causal analysis of parameter contributions, multi‑objective optimization, and systematic evaluation of the impact of freezing identified regions on downstream performance. This work serves as a reality check on the assumption that LLMs possess intrinsic, dataset‑agnostic safety zones and highlights the need for robust, reproducible safety‑region discovery techniques.