用扩散模型提升无数据图像生成质量,让合成图更像原始训练数据。
When Model Knowledge meets Diffusion Model: Diffusion-assisted Data-free Image Synthesis with Alignment of Domain and Class
- 利用文本到图像扩散模型作为图像先验,引导无数据图像生成。
- 在PACS和ImageNet上生成图像与真实数据分布对齐,性能达到新高。
- 适合研究无数据训练、模型知识迁移的学者使用。
开源预训练模型在诸多应用中潜力巨大,但当其训练数据不可用时,实用性下降。数据自由图像合成(DFIS)旨在不访问原始数据的情况下生成逼近预训练模型所学数据分布的图像。然而,现有方法因缺乏自然图像先验知识,生成样本偏离真实分布。为此,我们提出DDIS,首个利用扩散模型作为强大图像先验的扩散辅助数据自由图像合成方法,显著提升合成图像质量。DDIS从给定模型中提取知识,并用于指导扩散模型,使生成图像精准匹配训练数据分布。为此,我们引入领域对齐引导(DAG),在扩散采样过程中对齐合成数据域与训练数据域;同时优化单一类别对齐标记(CAT)嵌入,有效捕捉训练集中的类别特异性特征。在PACS与ImageNet上的实验表明,DDIS优于以往方法,生成样本更贴近真实数据分布,实现数据自由应用领域的最先进性能。
原文摘要 · Abstract (English)
Open-source pre-trained models hold great potential for diverse applications, but their utility declines when their training data is unavailable. Data-Free Image Synthesis (DFIS) aims to generate images that approximate the learned data distribution of a pre-trained model without accessing the original data. However, existing DFIS meth ods produce samples that deviate from the training data distribution due to the lack of prior knowl edge about natural images. To overcome this limitation, we propose DDIS, the first Diffusion-assisted Data-free Image Synthesis method that leverages a text-to-image diffusion model as a powerful image prior, improving synthetic image quality. DDIS extracts knowledge about the learned distribution from the given model and uses it to guide the diffusion model, enabling the generation of images that accurately align with the training data distribution. To achieve this, we introduce Domain Alignment Guidance (DAG) that aligns the synthetic data domain with the training data domain during the diffusion sampling process. Furthermore, we optimize a single Class Alignment Token (CAT) embedding to effectively capture class-specific attributes in the training dataset. Experiments on PACS and Ima geNet demonstrate that DDIS outperforms prior DFIS methods by generating samples that better reflect the training data distribution, achieving SOTA performance in data-free applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。