探究视觉语言模型对自身知识边界的认知能力，发现其自信判断常不靠谱。
Do LVLMs Know What They Know? A Systematic Study of Knowledge Boundary Perception in LVLMs
- 通过三种置信度信号评估模型是否清楚自己知道什么
- 概率和一致性置信度更可靠，口头表达的自信常导致过度自信
- 提出新方法提升模型自我认知，适合做可信AI的研究者参考
大型视觉语言模型（LVLMs）在视觉问答任务中表现强劲，但存在幻觉问题。可靠的模型应能识别自身的知识边界——知道自己知道什么、不知道什么。本文通过评估三种置信度信号：概率置信度、答案一致性置信度和口头表达置信度，系统研究了LVLMs对知识边界的感知能力。在三个LVLMs和三个VQA数据集上的实验表明，尽管模型具备一定感知能力，但仍存在较大提升空间。其中，概率和一致性置信度信号更为可靠，而口头置信度常导致过高的自信。为改善感知能力，我们借鉴大语言模型中的置信度校准方法，提出三种有效策略。此外，与纯语言模型相比，联合处理视觉与文本输入虽降低问答性能，但显著降低置信度水平，反而提升了知识边界的感知能力。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) demonstrate strong visual question answering (VQA) capabilities but are shown to hallucinate. A reliable model should perceive its knowledge boundaries-knowing what it knows and what it does not. This paper investigates LVLMs' perception of their knowledge boundaries by evaluating three types of confidence signals: probabilistic confidence, answer consistency-based confidence, and verbalized confidence. Experiments on three LVLMs across three VQA datasets show that, although LVLMs possess a reasonable perception level, there is substantial room for improvement. Among the three confidences, probabilistic and consistency-based signals are more reliable indicators, while verbalized confidence often leads to overconfidence. To enhance LVLMs' perception, we adapt several established confidence calibration methods from Large Language Models (LLMs) and propose three effective methods. Additionally, we compare LVLMs with their LLM counterparts, finding that jointly processing visual and textual inputs decreases question-answering performance but reduces confidence, resulting in an improved perception level compared to LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。