检测文生图模型是否盗用特定艺术家风格,无需修改原图或模型。
ArtistAuditor: Auditing Artist Style Pirate in Text-to-Image Generation Models
- 通过多粒度风格特征提取,识别模型是否微调过特定艺术家作品。
- 在6组实验中平均AUC超0.937,检测准确率高。
- 适用于已公开模型或图像的版权审计,适合艺术版权保护场景。
基于扩散过程的文生图模型(如DALL-E、Stable Diffusion、Midjourney)能将文本生成细节丰富的图像,在艺术与设计领域广泛应用。这导致业余用户可收集艺术家作品并微调模型,模仿专业画风,引发版权争议。现有方法分为两类:一类在作品中加入视觉不可见扰动(扰动法),另一类在作品中嵌入训练后可检测的水印(水印法)。但当作品或模型已在线发布时,无法修改原图或重新训练模型,这些方法失效。为此,我们提出ArtistAuditor,一种针对文生图模型的数据使用审计新方法。其核心思想是通过分析风格相关特征,判断可疑模型是否曾以特定艺术家的作品进行微调。具体而言,ArtistAuditor使用风格提取器获取多粒度风格表示,并将艺术作品视为艺术家风格的采样;随后通过训练好的判别器获得审计决策。在六组模型与数据集组合上的实验表明,ArtistAuditor的AUC值均超过0.937。通过对模型迁移性及核心模块的分析,我们提供了实践应用的洞见。最后,我们在一个真实在线平台场景中验证了该方法的有效性。代码已开源于https://github.com/Jozenn/ArtistAuditor。
原文摘要 · Abstract (English)
Text-to-image models based on diffusion processes, such as DALL-E, Stable Diffusion, and Midjourney, are capable of transforming texts into detailed images and have widespread applications in art and design. As such, amateur users can easily imitate professional-level paintings by collecting an artist's work and fine-tuning the model, leading to concerns about artworks' copyright infringement. To tackle these issues, previous studies either add visually imperceptible perturbation to the artwork to change its underlying styles (perturbation-based methods) or embed post-training detectable watermarks in the artwork (watermark-based methods). However, when the artwork or the model has been published online, i.e., modification to the original artwork or model retraining is not feasible, these strategies might not be viable. To this end, we propose a novel method for data-use auditing in the text-to-image generation model. The general idea of ArtistAuditor is to identify if a suspicious model has been finetuned using the artworks of specific artists by analyzing the features related to the style. Concretely, ArtistAuditor employs a style extractor to obtain the multi-granularity style representations and treats artworks as samplings of an artist's style. Then, ArtistAuditor queries a trained discriminator to gain the auditing decisions. The experimental results on six combinations of models and datasets show that ArtistAuditor can achieve high AUC values (> 0.937). By studying ArtistAuditor's transferability and core modules, we provide valuable insights into the practical implementation. Finally, we demonstrate the effectiveness of ArtistAuditor in real-world cases by an online platform Scenario. ArtistAuditor is open-sourced at https://github.com/Jozenn/ArtistAuditor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。