arXiv:2410.07336cs.CVcs.AI2024-10IJCV被引 20

用正样本增强对比学习,提升图文生成评估与训练效果。

Positive-Augmented Contrastive Learning for Vision-and-Language Evaluation and Training

  • 基于CLIP构建可学习度量,引入自生成图文正样本增强表征。
  • 在多个数据集上优于主流指标,能敏感检测对象幻觉问题。
  • 作为奖励用于自批判训练,使生成描述更丰富、少重复和语法错误。

尽管图像描述生成取得显著进展,现有评估指标仍难以捕捉描述质量或细粒度细节,主要因依赖非特定的人类撰写参考文本或噪声预训练数据。有效的评估指标不仅对评价至关重要,还能在生成阶段发挥关键作用。我们提出PAC-S++,一种基于预训练于网络收集与清洗数据的CLIP模型的可学习度量,通过额外生成的视觉-文本正样本对进行正则化。利用更强且更干净的预训练基础,我们将PAC-S++应用于自批判序列训练(SCST)阶段以微调描述模型。在多个图像与视频数据集上的大量实验表明,相比主流指标,PAC-S++表现更优,且对对象幻觉具有敏感性。此外,将PAC-S++融入微调阶段后,生成的描述语义更丰富,重复更少,语法错误更少。在跨域基准测试中也验证了该微调方法的有效性。源代码与训练模型公开于:https://github.com/aimagelab/pacscore。

原文摘要 · Abstract (English)

Despite significant advancements in caption generation, existing evaluation metrics often fail to capture the full quality or fine-grained details of captions. This is mainly due to their reliance on non-specific human-written references or noisy pre-training data. Still, finding an effective metric is crucial not only for captions evaluation but also for the generation phase. Metrics can indeed play a key role in the fine-tuning stage of captioning models, ultimately enhancing the quality of the generated captions. In this paper, we propose PAC-S++, a learnable metric that leverages the CLIP model, pre-trained on both web-collected and cleaned data and regularized through additional pairs of generated visual and textual positive samples. Exploiting this stronger and curated pre-training, we also apply PAC-S++ as a reward in the Self-Critical Sequence Training (SCST) stage typically employed to fine-tune captioning models. Extensive experiments on different image and video datasets highlight the effectiveness of PAC-S++ compared to popular metrics for the task, including its sensitivity to object hallucinations. Furthermore, we show that integrating PAC-S++ into the fine-tuning stage of a captioning model results in semantically richer captions with fewer repetitions and grammatical errors. Evaluations on out-of-domain benchmarks further demonstrate the efficacy of our fine-tuning approach in enhancing model capabilities. Source code and trained models are publicly available at: https://github.com/aimagelab/pacscore.

图文生成对比学习评估指标CLIP

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。