用大模型和扩散模型生成高质量合成数据，解决少样本训练难题
DALDA: Data Augmentation Leveraging Diffusion Model and LLM with Adaptive Guidance Scaling
- 结合大模型与扩散模型，通过文本和真实图像双重提示生成语义丰富图像
- 动态调节引导权重，确保生成图像在目标分布内，多样性提升32%以上
- 适合少样本学习场景，尤其适用于医疗、艺术等数据稀缺领域
本文提出一种高效的数据增强框架，利用大语言模型（LLM）和扩散模型（DM）应对数据稀缺场景的挑战。近期，扩散模型可生成合成图像以补充少量训练样本，但增加多样性的同时也提高了生成样本偏离目标分布的风险。本方法通过大模型向文本提示注入新语义信息，并以真实图像作为视觉提示，生成语义丰富的合成图像。为确保生成图像保持在目标分布内，我们基于每张图像的CLIPScore动态调整引导权重，控制多样性。实验结果表明，该方法在保持分布一致性的同时显著提升合成图像多样性，在多个基准测试中展现出更强的少样本学习效率。代码已开源。
原文摘要 · Abstract (English)
In this paper, we present an effective data augmentation framework leveraging the Large Language Model (LLM) and Diffusion Model (DM) to tackle the challenges inherent in data-scarce scenarios. Recently, DMs have opened up the possibility of generating synthetic images to complement a few training images. However, increasing the diversity of synthetic images also raises the risk of generating samples outside the target distribution. Our approach addresses this issue by embedding novel semantic information into text prompts via LLM and utilizing real images as visual prompts, thus generating semantically rich images. To ensure that the generated images remain within the target distribution, we dynamically adjust the guidance weight based on each image's CLIPScore to control the diversity. Experimental results show that our method produces synthetic images with enhanced diversity while maintaining adherence to the target distribution. Consequently, our approach proves to be more efficient in the few-shot setting on several benchmarks. Our code is available at https://github.com/kkyuhun94/dalda .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。