提出检测视觉大模型知识边界的方法，减少无效检索。
Detecting Knowledge Boundary of Vision Large Language Models by Sampling-Based Inference
- 基于采样推理构建数据集，微调模型识别知识边界。
- 在多类视觉问答数据集上提升检索效率，性能不降反升。
- 跨模型通用的知识边界可复用，适合优化RAG系统。
尽管视觉大语言模型（VLLM）取得进展，但与文本大模型类似，其在需要实时信息或知识密集型问题上仍存在局限。盲目采用检索增强生成（RAG）虽有效但成本高昂。为降低对检索的依赖，同时保持或提升检索带来的性能优势，本文提出一种检测VLLM知识边界的方法，实现更高效的RAG应用。具体地，提出两种变体方法，通过在自动构建的数据集上微调VLLM以识别知识边界。在多种视觉问答数据集上的实验表明，该方法能准确刻画模型的知识边界，据此可减少不必要的检索，同时维持甚至提升性能。此外，本方法为某一VLLM识别出的知识边界可作为其他VLLM的代理边界使用。代码将发布于https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary。
原文摘要 · Abstract (English)
Despite the advancements made in Vision Large Language Models (VLLMs), like text Large Language Models (LLMs), they have limitations in addressing questions that require real-time information or are knowledge-intensive. Indiscriminately adopting Retrieval Augmented Generation (RAG) techniques is an effective yet expensive way to enable models to answer queries beyond their knowledge scopes. To mitigate the dependence on retrieval and simultaneously maintain, or even improve, the performance benefits provided by retrieval, we propose a method to detect the knowledge boundary of VLLMs, allowing for more efficient use of techniques like RAG. Specifically, we propose a method with two variants that fine-tune a VLLM on an automatically constructed dataset for boundary identification. Experimental results on various types of Visual Question Answering datasets show that our method successfully depicts a VLLM's knowledge boundary, based on which we are able to reduce indiscriminate retrieval while maintaining or improving the performance. In addition, we show that the knowledge boundary identified by our method for one VLLM can be used as a surrogate boundary for other VLLMs. Code will be released at https://github.com/Chord-Chen-30/VLLM-KnowledgeBoundary
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。