arXiv:2505.10551cs.CVcs.AI2025-05被引 5

研究发现,合成数据的现实可行性对CLIP分类器影响很小。

Does Feasibility Matter? Understanding the Impact of Feasibility on Synthetic Training Data

  • 用可控编辑生成具可行或不可行属性的合成图像
  • 可行性差异对准确率影响小于0.3%,三数据集均如此
  • 混合使用可行与不可行数据效果稳定,适合实际应用

随着逼真扩散模型的发展,基于部分或全部合成数据训练的模型表现持续提升。然而,扩散模型常生成现实中不存在的图像,如漂浮的狗或纹理失真。本文定义‘可行性’为合成图像中属性在真实世界中是否可能存在的准则;不符合此准则的图像被视为不可行。直观上,不可行图像属于分布外数据,训练时应排除。但可行性真的重要吗?本文研究在基于CLIP的分类器中,生成合成训练数据时强制可行性是否必要,聚焦背景、颜色和纹理三种属性。提出VariReal框架,通过最小编辑将源图像调整为符合大语言模型提示的可行或不可行属性。实验表明,可行性对LoRA微调后CLIP性能影响极小,三细粒度数据集上top-1准确率差异普遍低于0.3%。且属性类型决定了可行/不可行图像是否会产生负面影响。此外,混合使用可行与不可行图像的训练效果与纯可行或纯不可行数据集相比无显著差异。

原文摘要 · Abstract (English)

With the development of photorealistic diffusion models, models trained in part or fully on synthetic data achieve progressively better results. However, diffusion models still routinely generate images that would not exist in reality, such as a dog floating above the ground or with unrealistic texture artifacts. We define the concept of feasibility as whether attributes in a synthetic image could realistically exist in the real-world domain; synthetic images containing attributes that violate this criterion are considered infeasible. Intuitively, infeasible images are typically considered out-of-distribution; thus, training on such images is expected to hinder a model's ability to generalize to real-world data, and they should therefore be excluded from the training set whenever possible. However, does feasibility really matter? In this paper, we investigate whether enforcing feasibility is necessary when generating synthetic training data for CLIP-based classifiers, focusing on three target attributes: background, color, and texture. We introduce VariReal, a pipeline that minimally edits a given source image to include feasible or infeasible attributes given by the textual prompt generated by a large language model. Our experiments show that feasibility minimally affects LoRA-fine-tuned CLIP performance, with mostly less than 0.3% difference in top-1 accuracy across three fine-grained datasets. Also, the attribute matters on whether the feasible/infeasible images adversarially influence the classification performance. Finally, mixing feasible and infeasible images in training datasets does not significantly impact performance compared to using purely feasible or infeasible datasets.

合成数据扩散模型可行性CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。