通过目标感知提示减少图像描述中的虚构对象
OPCap:Object-aware Prompting Captioning
- 用检测器提取物体标签与位置，再用属性预测器优化语义特征
- 在COCO和nocaps数据集上显著降低幻觉率，提升描述质量
- 适合关注生成准确性与真实性的视觉语言模型研究者
在图像描述任务中，缺失或不存在的物体被用来解释图像的现象称为对象偏差（或幻觉）。为缓解此问题，我们提出一种目标感知提示策略。该方法首先利用物体检测器从图像中提取物体标签及其空间信息，随后通过属性预测器进一步优化物体的语义特征。这些优化后的特征被整合并输入解码器，增强模型对图像上下文的理解。在COCO和nocaps数据集上的实验结果表明，OPCap能有效缓解幻觉现象，并显著提升生成描述的质量。
原文摘要 · Abstract (English)
In the field of image captioning, the phenomenon where missing or nonexistent objects are used to explain an image is referred to as object bias (or hallucination). To mitigate this issue, we propose a target-aware prompting strategy. This method first extracts object labels and their spatial information from the image using an object detector. Then, an attribute predictor further refines the semantic features of the objects. These refined features are subsequently integrated and fed into the decoder, enhancing the model's understanding of the image context. Experimental results on the COCO and nocaps datasets demonstrate that OPCap effectively mitigates hallucination and significantly improves the quality of generated captions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。