让视觉语言模型学会根据对话上下文准确引用相关图片。
ImageRef-VL: Enabling Contextual Image Referencing in Vision-Language Models
- 通过指令微调提升模型在对话中引用相关图像的能力。
- 在新构建的数据集上，性能比现有开源模型提升88%。
- 适合需要图文联动的智能客服、多模态问答系统使用。
视觉语言模型（VLMs）在理解多模态输入方面展现出强大能力，并被广泛应用于基于检索增强生成（RAG）的对话系统中。尽管当前VLM驱动的聊天机器人能在回答中提供文本来源引用，但在对话过程中准确引用相关图片方面仍存在显著局限。本文首次提出“上下文图像引用”能力——即根据对话上下文从检索文档中恰当引用相关图像，并系统评估了VLM在这方面的表现。我们构建了首个专门用于上下文图像引用的测试数据集与评估指标。此外，我们提出了ImageRef-VL方法，通过在大规模人工标注的多模态对话数据集上进行指令微调，显著提升了开源VLM的图像引用能力。实验结果表明，ImageRef-VL不仅优于专有模型，还在上下文图像引用任务中相较当前最佳开源VLM实现88%的性能提升。代码已开源：https://github.com/bytedance/ImageRef-VL。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have demonstrated remarkable capabilities in understanding multimodal inputs and have been widely integrated into Retrieval-Augmented Generation (RAG) based conversational systems. While current VLM-powered chatbots can provide textual source references in their responses, they exhibit significant limitations in referencing contextually relevant images during conversations. In this paper, we introduce Contextual Image Reference -- the ability to appropriately reference relevant images from retrieval documents based on conversation context -- and systematically investigate VLMs' capability in this aspect. We conduct the first evaluation for contextual image referencing, comprising a dedicated testing dataset and evaluation metrics. Furthermore, we propose ImageRef-VL, a method that significantly enhances open-source VLMs' image referencing capabilities through instruction fine-tuning on a large-scale, manually curated multimodal conversation dataset. Experimental results demonstrate that ImageRef-VL not only outperforms proprietary models but also achieves an 88% performance improvement over state-of-the-art open-source VLMs in contextual image referencing tasks. Our code is available at https://github.com/bytedance/ImageRef-VL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。