提出可抵抗微调的文本生成图像模型水印技术,保护知识产权。
SleeperMark: Towards Robust Watermark against Fine-Tuning Text-to-image Diffusion Models
- 通过解耦语义与水印信息,让模型在微调时仍保留水印
- 在多种扩散模型上验证,微调后仍能有效识别水印
- 适合需要保护生成模型版权的研究者和开发者
近期大规模文本到图像(T2I)扩散模型的发展推动了风格定制、主体个性化及条件生成等下游应用。由于训练需大量数据和算力,这些模型构成高价值知识产权,易遭恶意微调以牟利。现有水印方法依赖生成结果或特征空间验证,但在模型微调后失效,且黑盒场景下特征空间不可访问。模型在适应新任务时容易遗忘原有水印。为此,我们提出SleeperMark框架,通过显式引导模型将水印信息与语义概念解耦,使模型在持续微调过程中仍能保持水印。实验表明,SleeperMark在多种扩散模型(如Stable Diffusion、DeepFloyd-IF)上对下游微调及图像级、模型级攻击均具鲁棒性,且对生成能力影响极小。代码已开源。
原文摘要 · Abstract (English)
Recent advances in large-scale text-to-image (T2I) diffusion models have enabled a variety of downstream applications, including style customization, subject-driven personalization, and conditional generation. As T2I models require extensive data and computational resources for training, they constitute highly valued intellectual property (IP) for their legitimate owners, yet making them incentive targets for unauthorized fine-tuning by adversaries seeking to leverage these models for customized, usually profitable applications. Existing IP protection methods for diffusion models generally involve embedding watermark patterns and then verifying ownership through generated outputs examination, or inspecting the model's feature space. However, these techniques are inherently ineffective in practical scenarios when the watermarked model undergoes fine-tuning, and the feature space is inaccessible during verification ((i.e., black-box setting). The model is prone to forgetting the previously learned watermark knowledge when it adapts to a new task. To address this challenge, we propose SleeperMark, a novel framework designed to embed resilient watermarks into T2I diffusion models. SleeperMark explicitly guides the model to disentangle the watermark information from the semantic concepts it learns, allowing the model to retain the embedded watermark while continuing to be adapted to new downstream tasks. Our extensive experiments demonstrate the effectiveness of SleeperMark across various types of diffusion models, including latent diffusion models (e.g., Stable Diffusion) and pixel diffusion models (e.g., DeepFloyd-IF), showing robustness against downstream fine-tuning and various attacks at both the image and model levels, with minimal impact on the model's generative capability. The code is available at https://github.com/taco-group/SleeperMark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。