DSP-Reg: Domain-Sensitive Parameter Regularization for Robust Domain Generalization

Domain Generalization (DG) is a critical area that focuses on developing models capable of performing well on data from unseen distributions, which is essential for real-world applications. Existing approaches primarily concentrate on learning domain-invariant features, which assume that a model robust to variations in the source domains will generalize well to unseen target domains. However, these approaches neglect a deeper analysis at the parameter level, which makes the model hard to explicitly differentiate between parameters sensitive to domain shifts and those robust, potentially hindering its overall ability to generalize. In order to address these limitations, we first build a covariance-based parameter sensitivity analysis framework to quantify the sensitivity of each parameter in a model to domain shifts. By computing the covariance of parameter gradients across multiple source domains, we can identify parameters that are more susceptible to domain variations, which serves as our theoretical foundation. Based on this, we propose Domain-Sensitive Parameter Regularization (DSP-Reg), a principled framework that guides model optimization by a soft regularization technique that encourages the model to rely more on domain-invariant parameters while suppressing those that are domain-specific. This approach provides a more granular control over the model’s learning process, leading to improved robustness and generalization to unseen domains. Extensive experiments on benchmarks, such as PACS, VLCS, OfficeHome, and DomainNet, demonstrate that DSP-Reg outperforms state-of-the-art approaches, achieving an average accuracy of 66.7% and surpassing all baselines.

💡 Research Summary

Domain Generalization (DG) seeks models that maintain high performance when deployed on unseen, out‑of‑distribution target domains. Most existing DG methods operate at the feature level, encouraging domain‑invariant representations through adversarial training, discrepancy minimization, meta‑learning, or data augmentation. While effective, these approaches overlook the granularity that parameter‑level analysis can provide: they cannot explicitly distinguish parameters that are robust across domains from those that are highly sensitive to domain shifts.

The paper introduces Domain‑Sensitive Parameter Regularization (DSP‑Reg), a novel framework that first quantifies the sensitivity of each network parameter to domain variations and then uses this information to guide training. The authors construct a covariance‑based sensitivity analysis: they linearize the network around a nominal point, model input and parameter perturbations as zero‑mean Gaussian variables with covariances Σ_x and Σ_θ, and propagate second‑order moments through the linearized relation δy ≈ J_x δx + J_θ δθ. This yields an output covariance consisting of an input‑induced term and a parameter‑induced term J_θ Σ_θ J_θᵀ. Assuming Σ_θ is diagonal, the parameter‑induced variance decomposes into a sum over individual parameters: Var_output = Σ_k Var(θ_k) ‖∂θ_k f(x)‖².

From this decomposition the authors define a per‑parameter sensitivity index
s_k = Var(θ_k) · E_x