arXiv:2605.14291cs.CRcs.AI2026-05被引 2

通过微扰生成不可学习数据,主动防范视觉语言模型非法训练

To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model

论文配图:To See is Not to Learn: Protecting Multimodal Data from Unauthorized Fine-Tuning of Large Vision-Language Model
图 1 · 摘自论文原文
  • 在图像中注入人眼不可察觉的干扰,让模型学坏
  • 使模型在真实数据上性能下降超过30%(在COCO上)
  • 适合关注版权保护的研究者与数据拥有者

大型视觉语言模型(LVLM)的快速发展伴随着对多模态网络数据的未经授权抓取和训练,严重威胁数据所有者的版权与隐私。现有对策如机器遗忘和水印均为事后手段,仅在侵权发生后起效。本文提出MMGuard,使数据所有者能主动防御未经许可的LVLM微调。该方法通过注入人眼不可察觉的扰动,利用LVLM的学习动态生成不可学习样本。扰动最小化训练损失,制造优化捷径,导致模型过拟合噪声,推理时无扰动情况下性能显著下降。为进一步强化防御,MMGuard引入跨模态绑定破坏机制,策略性地引导模型注意力,建立噪声与目标间的虚假关联,并有理论保证。结合集成学习策略提升跨模型迁移能力。在六大数据集上对九个开源LVLM进行评估,结果表明,在白盒、灰盒和黑盒攻击模型下均具有效性、隐蔽性和鲁棒性,建立了主动防御的机制优势。

原文摘要 · Abstract (English)

The rapid advancement of Large Vision-Language Models (LVLMs) is increasingly accompanied by unauthorized scraping and training on multimodal web data, posing severe copyright and privacy risks to data owners. Existing countermeasures, such as machine unlearning and watermarks, are inherent post-hoc approaches that act only after intellectual property infringement has already occurred. In this work, we propose MMGuard to empower data owners to proactively protect their multimodal data against unauthorized LVLM fine-tuning. MMGuard generates unlearnable examples by injecting human-imperceptible perturbations that actively exploit the learning dynamics of LVLMs. By minimizing the training loss, the perturbation creates an optimization shortcut, causing the model to overfit to the noise and thereby degrading downstream performance when the perturbation is absent during inference. To further strengthen this defense, MMGuard introduces a cross-modal binding disruption, strategically shifting LVLM attention to enforce a spurious correlation between the noise and the training target with theoretical guarantees. Enhanced by an ensemble learning strategy for cross-model transferability, MMGuard is evaluated against nine open-source LVLMs across six datasets. Our comprehensive results demonstrate effective, stealthy, and robust protection under white-box, gray-box, and black-box threat models, establishing a mechanistic advantage in proactively defending against aggressive fine-tuning exploitation.

版权保护视觉语言模型对抗扰动

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。