arXiv:2503.10081cs.CVcs.CR2025-03ICLR被引 12

用对抗扰动干扰扩散模型的修复能力,防止恶意篡改图像内容。

AdvPaint: Protecting Images from Inpainting Manipulation via Adversarial Attention Disruption

  • 针对扩散模型的修复任务,通过干扰注意力机制来破坏语义理解。
  • 在多种掩码形状下,FID提升超100点,精度显著下降。
  • 适合关注图像安全防护的研究者和开发者使用。

扩散模型生成高质量图像的能力若被恶意利用,将带来严重威胁。本文假设攻击者利用扩散模型进行图像修复(如替换特定区域为名人),而现有防御方法多聚焦于图像到图像或文本到图像生成任务,对未经授权的修复行为防护不足,导致效果不佳。为此,我们提出ADVPAINT,一种新型防御框架,通过生成对抗扰动,有效干扰目标扩散修复模型的自注意力与交叉注意力模块,扰乱生成过程中的语义理解与提示交互。ADVPAINT采用两阶段扰动策略,基于目标对象的扩大边界框划分扰动区域,提升对不同形状和大小掩码的鲁棒性。实验表明,该方法在多项指标上优于现有方法:FID得分提升超过100点,精度显著降低,能有效阻止恶意修复行为。

原文摘要 · Abstract (English)

The outstanding capability of diffusion models in generating high-quality images poses significant threats when misused by adversaries. In particular, we assume malicious adversaries exploiting diffusion models for inpainting tasks, such as replacing a specific region with a celebrity. While existing methods for protecting images from manipulation in diffusion-based generative models have primarily focused on image-to-image and text-to-image tasks, the challenge of preventing unauthorized inpainting has been rarely addressed, often resulting in suboptimal protection performance. To mitigate inpainting abuses, we propose ADVPAINT, a novel defensive framework that generates adversarial perturbations that effectively disrupt the adversary's inpainting tasks. ADVPAINT targets the self- and cross-attention blocks in a target diffusion inpainting model to distract semantic understanding and prompt interactions during image generation. ADVPAINT also employs a two-stage perturbation strategy, dividing the perturbation region based on an enlarged bounding box around the object, enhancing robustness across diverse masks of varying shapes and sizes. Our experimental results demonstrate that ADVPAINT's perturbations are highly effective in disrupting the adversary's inpainting tasks, outperforming existing methods; ADVPAINT attains over a 100-point increase in FID and substantial decreases in precision.

图像安全对抗防御扩散模型修复防护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。