解决图像描述过早结束问题，让模型生成更长更详细的描述。
The Devil is in the EOS: Sequence Training for Detailed Image Captioning
- 通过无监督方法削弱模型对结束符的过早预测倾向。
- 在三个基准上显著提升描述长度与细节，但幻觉率略有上升。
- 无需复杂奖励或标注，适配任意预训练视觉语言模型。
尽管视觉语言模型（VLMs）取得了显著进展，图像描述仍常缺乏细节，基础模型生成的描述短且通用。这一问题在拥有强大视觉与语言主干网络的情况下依然存在。虽然已有研究尝试使用监督数据和复杂奖励函数提升细节生成，但我们发现根本原因在于交叉熵训练中引入的对序列结束（EOS）标记的偏好。为此，我们提出一种无监督方法，减轻模型过早预测EOS的倾向。通过降低该偏差，我们促使模型生成更长、更详细的描述，而无需复杂的奖励函数或监督信号。该方法简单有效，可应用于任意预训练模型。实验在三种VLM和三个详细描述基准上验证了其有效性，结果显示描述长度和相关细节明显提升，但幻觉率有所增加。
原文摘要 · Abstract (English)
Despite significant advances in vision-language models (VLMs), image captioning often suffers from a lack of detail, with base models producing short, generic captions. This limitation persists even though VLMs are equipped with strong vision and language backbones. While supervised data and complex reward functions have been proposed to improve detailed image captioning, we identify a simpler underlying issue: a bias towards the end-of-sequence (EOS) token, which is introduced during cross-entropy training. We propose an unsupervised method to debias the model's tendency to predict the EOS token prematurely. By reducing this bias, we encourage the generation of longer, more detailed captions without the need for intricate reward functions or supervision. Our approach is straightforward, effective, and easily applicable to any pretrained model. We demonstrate its effectiveness through experiments with three VLMs and on three detailed captioning benchmarks. Our results show a substantial increase in caption length and relevant details, albeit with an expected increase in the rate of hallucinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。