用强化学习让图像描述更精准，减少视觉信息丢失。
Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learning
- 通过文本检索匹配图像来评估描述质量，构建无标注训练框架。
- 在COCO-LN500上使关系推理能力提升20%（Qwen2.5-VL-7B）。
- 无需额外标注，适合提升大模型图像理解与生成能力。
大型视觉语言模型在生成图像描述时，常会遗漏或错误表达关键视觉内容。为减少此类信息损失，需促使模型关注图像细节以生成精确描述。然而，由于视觉内容与文本输出之间的模态差异，衡量模态转换中的信息损失极具挑战性。本文提出基于文本检索匹配的评估思路：图像描述的质量与其生成后通过文本搜索召回的图像相关性正相关。据此，我们设计了跨模态身份映射（Cross-modal Identity Mapping, CIM），一种无需额外标注的强化学习框架。该方法从两个维度量化信息损失：图库表征一致性与查询-图库图像相关性。在上述指标监督下，模型最小化信息损失，实现图像到描述的等价映射。实验表明，该方法在图像描述任务中表现优异，甚至优于有监督微调。尤其在COVO-LN500基准上，Qwen2.5-VL-7B模型的关系推理能力提升20%。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) often omit or misrepresent critical visual content in generated image captions. Minimizing such information loss will force LVLMs to focus on image details to generate precise descriptions. However, measuring information loss during modality conversion is inherently challenging due to the modal gap between visual content and text output. In this paper, we argue that the quality of an image caption is positively correlated with the similarity between images retrieved via text search using that caption. Based on this insight, we further propose Cross-modal Identity Mapping (CIM), a reinforcement learning framework that enhances image captioning without requiring additional annotations. Specifically, the method quantitatively evaluates the information loss from two perspectives: Gallery Representation Consistency and Query-gallery Image Relevance. Supervised under these metrics, LVLM minimizes information loss and aims to achieve identity mapping from images to captions. The experimental results demonstrate the superior performance of our method in image captioning, even when compared with Supervised Fine-Tuning. Particularly, on the COCO-LN500 benchmark, CIM achieves a 20% improvement in relation reasoning on Qwen2.5-VL-7B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。