arXiv:2602.23952cs.CV2026-02中稿 · CVPR被引 2

解决视觉问答中模型知识与外部信息冲突问题,提升准确率。

CC-VQA: Conflict- and Correlation-Aware Method for Mitigating Knowledge Conflict in Knowledge-Based Visual Question Answering

  • 从视觉语义角度分析内部与外部知识的冲突
  • 通过相关性引导编码解码,提升冲突识别精度
  • 无需训练,适合追求高精度的视觉问答研究者

基于知识的视觉问答(KB-VQA)在处理知识密集型任务方面具有巨大潜力。然而,由于视觉语言模型(VLMs)预训练时的静态参数化知识与动态检索信息之间存在冲突,导致输出要么忽略检索内容,要么与参数知识不一致,严重制约了KB-VQA的发展。现有缓解方法多借鉴语言模型策略,仅关注上下文层面的冲突,忽视视觉信息的作用,且易受冗余检索内容干扰,影响冲突识别与缓解效果。为此,本文提出一种无需训练的新型方法CC-VQA,包含两个核心组件:(1) 视觉中心的上下文冲突推理,实现对内部与外部知识上下文的视觉-语义冲突分析;(2) 相关性引导的编码与解码机制,采用位置编码压缩处理低相关性陈述,并结合相关性加权冲突评分进行自适应解码。在E-VQA、InfoSeek和OK-VQA三个基准上的大量实验表明,CC-VQA达到当前最优性能,相较已有方法绝对准确率提升3.3%至6.4%。代码已开源。

原文摘要 · Abstract (English)

Knowledge-based visual question answering (KB-VQA) demonstrates significant potential for handling knowledge-intensive tasks. However, conflicts arise between static parametric knowledge in vision language models (VLMs) and dynamically retrieved information due to the static model knowledge from pre-training. The outputs either ignore retrieved contexts or exhibit inconsistent integration with parametric knowledge, posing substantial challenges for KB-VQA. Current knowledge conflict mitigation methods primarily adapted from language-based approaches, focusing on context-level conflicts through engineered prompting strategies or context-aware decoding mechanisms. However, these methods neglect the critical role of visual information in conflicts and suffer from redundant retrieved contexts, which impair accurate conflict identification and effective mitigation. To address these limitations, we propose \textbf{CC-VQA}: a novel training-free, conflict- and correlation-aware method for KB-VQA. Our method comprises two core components: (1) Vision-Centric Contextual Conflict Reasoning, which performs visual-semantic conflict analysis across internal and external knowledge contexts; and (2) Correlation-Guided Encoding and Decoding, featuring positional encoding compression for low-correlation statements and adaptive decoding using correlation-weighted conflict scoring. Extensive evaluations on E-VQA, InfoSeek, and OK-VQA benchmarks demonstrate that CC-VQA achieves state-of-the-art performance, yielding absolute accuracy improvements of 3.3\% to 6.4\% compared to existing methods. Code is available at https://github.com/cqu-student/CC-VQA.

视觉问答知识冲突多模态模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。