用图像重建提升图像描述的准确性和完整性。
RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction
- 通过文本生成图像并比对差异来迭代优化描述。
- 在CapsBench和CompreCap上比多数基线提升约10%。
- 适合需要高质量图文数据的多模态任务研究者。
图像重描述广泛用于为多种多模态任务生成高质量训练数据。现有方法通常依赖强大的多模态大语言模型（MLLM）增强文本描述，但常因幻觉和遗漏细粒度细节导致不准确与不完整。为此，我们提出RICO框架，通过视觉重建精炼描述。具体而言，利用文本到图像模型将描述重构为参考图像，并提示MLLM识别原始与重构图像间的差异以优化描述。该过程迭代进行，逐步生成更忠实、更全面的描述。为缓解迭代带来的额外计算开销，我们引入RICO-Flash，通过DPO学习生成类似RICO的描述。大量实验表明，该方法显著提升描述的准确性和完整性，在CapsBench和CompreCap上优于大多数基线约10%。代码已开源：https://github.com/wangyuchi369/RICO。
原文摘要 · Abstract (English)
Image recaptioning is widely used to generate training datasets with enhanced quality for various multimodal tasks. Existing recaptioning methods typically rely on powerful multimodal large language models (MLLMs) to enhance textual descriptions, but often suffer from inaccuracies due to hallucinations and incompleteness caused by missing fine-grained details. To address these limitations, we propose RICO, a novel framework that refines captions through visual reconstruction. Specifically, we leverage a text-to-image model to reconstruct a caption into a reference image, and prompt an MLLM to identify discrepancies between the original and reconstructed images to refine the caption. This process is performed iteratively, further progressively promoting the generation of more faithful and comprehensive descriptions. To mitigate the additional computational cost induced by the iterative process, we introduce RICO-Flash, which learns to generate captions like RICO using DPO. Extensive experiments demonstrate that our approach significantly improves caption accuracy and completeness, outperforms most baselines by approximately 10% on both CapsBench and CompreCap. Code released at https://github.com/wangyuchi369/RICO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。