让AI生成答案时精准标注证据位置，提升可验证性。
VISA: Retrieval Augmented Generation with Visual Source Attribution
- 用视觉语言模型定位答案依据的文档区域并加框标注。
- 在维基和医学文献数据集上验证，能准确识别支持性图文区域。
- 适合需要可追溯答案来源的研究与医疗问答场景。
带溯源的生成对提升检索增强生成（RAG）系统的可验证性至关重要。然而现有RAG方法多仅关联文档级引用，用户难以在内容丰富的多文档中定位证据。为此，我们提出视觉溯源增强生成（VISA），结合答案生成与视觉溯源。借助大视觉语言模型（VLMs），VISA在检索到的文档截图中识别支持性证据，并以边界框精确标出对应区域。为评估效果，我们构建了两个数据集：基于爬取维基网页截图的Wiki-VISA，以及源自PubLayNet、面向医疗领域的Paper-VISA。实验表明，VISA在保持文档原始外观的前提下，有效实现视觉溯源，同时揭示了改进空间。代码、数据与模型检查点将公开。
原文摘要 · Abstract (English)
Generation with source attribution is important for enhancing the verifiability of retrieval-augmented generation (RAG) systems. However, existing approaches in RAG primarily link generated content to document-level references, making it challenging for users to locate evidence among multiple content-rich retrieved documents. To address this challenge, we propose Retrieval-Augmented Generation with Visual Source Attribution (VISA), a novel approach that combines answer generation with visual source attribution. Leveraging large vision-language models (VLMs), VISA identifies the evidence and highlights the exact regions that support the generated answers with bounding boxes in the retrieved document screenshots. To evaluate its effectiveness, we curated two datasets: Wiki-VISA, based on crawled Wikipedia webpage screenshots, and Paper-VISA, derived from PubLayNet and tailored to the medical domain. Experimental results demonstrate the effectiveness of VISA for visual source attribution on documents' original look, as well as highlighting the challenges for improvement. Code, data, and model checkpoints will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。