通过平衡正负文本分布，提升图文对齐能力
Removing Distributional Discrepancies in Captions Improves Image-Text Alignment
- 用正样本生成混合类型负样本，改善训练数据分布
- 在多个数据集上超越现有最优方法，显著提升对齐精度
- 适合需要精准图文匹配的生成模型评估与优化
本文提出一种改进图像-文本对齐预测能力的模型，针对当前视觉-语言模型在组合理解上的挑战。方法核心是通过正样本生成混合类型的负样本，构建高质量训练数据，并解决正负样本间的分布不平衡问题，确保模型不仅依赖文本信息，还能结合图像内容准确预测对齐关系。在此基础上，对现有领先视觉-语言模型进行微调，显著提升了其对齐理解能力。实验表明，该方法在多个数据集上优于当前最佳方法。此外，还验证了模型在基于文本生成图像的质量排序中的应用价值。
原文摘要 · Abstract (English)
In this paper, we introduce a model designed to improve the prediction of image-text alignment, targeting the challenge of compositional understanding in current visual-language models. Our approach focuses on generating high-quality training datasets for the alignment task by producing mixed-type negative captions derived from positive ones. Critically, we address the distribution imbalance between positive and negative captions to ensure that the alignment model does not depend solely on textual information but also considers the associated images for predicting alignment accurately. By creating this enhanced training data, we fine-tune an existing leading visual-language model to boost its capability in understanding alignment. Our model significantly outperforms current top-performing methods across various datasets. We also demonstrate the applicability of our model by ranking the images generated by text-to-image models based on text alignment. Project page: \url{https://yuheng-li.github.io/LLaVA-score/}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。