PoGDiff用高斯乘积改善文本到图像生成中的数据不平衡问题
PoGDiff: Product-of-Gaussians Diffusion Models for Imbalanced Text-to-Image Generation
- 用高斯乘积替代真实分布，缓解少数类样本不足问题
- 在真实数据集上提升生成准确率与图像质量
- 适合处理标注不均衡的文本图像生成任务
扩散模型近年来取得显著进展，但在不平衡数据集上训练或微调时性能往往下降。这主要源于图像-文本对中多数类与少数类样本比例失衡。本文提出一种通用微调方法PoGDiff，解决此问题。不同于直接最小化预测分布与真实分布间的KL散度，PoGDiff将真实分布替换为由原始目标与邻近文本嵌入条件下的预测分布组合而成的高斯乘积（PoG）。在真实世界数据集上的实验表明，该方法有效缓解了扩散模型中的数据不平衡问题，提升了生成准确率与图像质量。
原文摘要 · Abstract (English)
Diffusion models have made significant advancements in recent years. However, their performance often deteriorates when trained or fine-tuned on imbalanced datasets. This degradation is largely due to the disproportionate representation of majority and minority data in image-text pairs. In this paper, we propose a general fine-tuning approach, dubbed PoGDiff, to address this challenge. Rather than directly minimizing the KL divergence between the predicted and ground-truth distributions, PoGDiff replaces the ground-truth distribution with a Product of Gaussians (PoG), which is constructed by combining the original ground-truth targets with the predicted distribution conditioned on a neighboring text embedding. Experiments on real-world datasets demonstrate that our method effectively addresses the imbalance problem in diffusion models, improving both generation accuracy and quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。