Visual-Friendly Concept Protection via Selective Adversarial Perturbations

Personalized concept generation by tuning diffusion models with a few images raises potential legal and ethical concerns regarding privacy and intellectual property rights. Researchers attempt to prevent malicious personalization using adversarial perturbations. However, previous efforts have mainly focused on the effectiveness of protection while neglecting the visibility of perturbations. They utilize global adversarial perturbations, which introduce noticeable alterations to original images and significantly degrade visual quality. In this work, we propose the Visual-Friendly Concept Protection (VCPro) framework, which prioritizes the protection of key concepts chosen by the image owner through adversarial perturbations with lower perceptibility. To ensure these perturbations are as inconspicuous as possible, we introduce a relaxed optimization objective to identify the least perceptible yet effective adversarial perturbations, solved using the Lagrangian multiplier method. Qualitative and quantitative experiments validate that VCPro achieves a better trade-off between the visibility of perturbations and protection effectiveness, effectively prioritizing the protection of target concepts in images with less perceptible perturbations.

💡 Research Summary

**
The paper addresses the emerging privacy and intellectual‑property risks posed by personalized diffusion models such as Stable Diffusion, which can learn a new concept from only a few reference images. Existing defenses rely on adversarial perturbations that are applied globally to the whole image. While they can hinder unauthorized personalization (e.g., DreamBooth, Textual Inversion), the perturbations are typically large (11/255–17/255) and produce visible artifacts that degrade user experience, especially for sensitive content like face photos.

VCPro (Visual‑Friendly Concept Protection) proposes a fundamentally different approach: protect only the user‑specified “important” regions (e.g., a face) and keep the rest of the image untouched. Users supply a binary mask m either manually or via segmentation tools such as SAM. The protection objective is formulated as a regional adversarial loss that contains a “push” term (maximizing the distance between predicted and ground‑truth noise in masked regions) and a “pull” term (minimizing that distance in unmasked regions). This dual loss forces the diffusion model’s denoising process to fail on the protected region while preserving visual fidelity elsewhere.

To further reduce perceptibility, the authors observe that the human visual system is more sensitive to low‑frequency changes. They therefore constrain perturbations to high‑frequency components by applying a discrete wavelet transform (DWT). The distance metric D(·) used in the optimization is computed only on the high‑frequency sub‑bands, effectively hiding the noise where the eye is less likely to notice it.

The optimization problem is relaxed into a Lagrangian‑multiplier formulation: minimize c·D(x, x+δ) − L′θ + α subject to δ ∈