arXiv:2410.03039cs.CVcs.AI2024-10ICML被引 1

通过模型引导反推微调数据,可从公开的个性化扩散模型中提取约20%训练图像。

Leveraging Model Guidance to Extract Training Data from Personalized Diffusion Models

  • 将微调视为分布渐变,用外推引导生成高概率图像。
  • 在多个数据集上成功提取约20%的原始训练数据。
  • 适合关注模型隐私与版权风险的研究者或开发者。

扩散模型(DMs)已成为强大的图像生成工具,尤其在少样本微调场景中,通过少量图像对预训练模型进行微调以捕捉特定风格或对象。许多用户将此类个性化模型上传至Civitai、HuggingFace等平台,形成活跃社区。然而,模型发布者可能忽视了数据泄露风险,且未经授权使用数据微调也引发版权争议。本文提出:能否从公开共享的微调模型中提取训练数据?若可行,不仅带来数据泄露威胁,还可提供侵权证据。为此,我们提出FineXtract框架,将微调过程建模为模型学习分布的渐进变化——从预训练模型分布向微调数据分布迁移。通过外推微调前后的模型,引导生成聚焦于微调数据分布的高概率区域,并结合聚类算法提取最可能的图像。在WikiArt、DreamBooth及真实在线模型上的实验表明,该方法在多数情况下可成功提取约20%的原始训练数据。代码已开源:https://github.com/Nicholas0228/FineXtract。

原文摘要 · Abstract (English)

Diffusion Models (DMs) have become powerful image generation tools, especially for few-shot fine-tuning where a pretrained DM is fine-tuned on a small image set to capture specific styles or objects. Many people upload these personalized checkpoints online, fostering communities such as Civitai and HuggingFace. However, model owners may overlook the data leakage risks when releasing fine-tuned checkpoints. Moreover, concerns regarding copyright violations arise when unauthorized data is used during fine-tuning. In this paper, we ask: "Can training data be extracted from these fine-tuned DMs shared online?" A successful extraction would present not only data leakage threats but also offer tangible evidence of copyright infringement. To answer this, we propose FineXtract, a framework for extracting fine-tuning data. Our method approximates fine-tuning as a gradual shift in the model's learned distribution -- from the original pretrained DM toward the fine-tuning data. By extrapolating the models before and after fine-tuning, we guide the generation toward high-probability regions within the fine-tuned data distribution. We then apply a clustering algorithm to extract the most probable images from those generated using this extrapolated guidance. Experiments on DMs fine-tuned with datasets including WikiArt, DreamBooth, and real-world checkpoints posted online validate the effectiveness of our method, extracting about 20% of fine-tuning data in most cases. The code is available https://github.com/Nicholas0228/FineXtract.

扩散模型数据提取版权风险

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。