通过加权多相关上下文,提升视觉问答的精准度。
Relevance-aware Multi-context Contrastive Decoding for Retrieval-augmented Visual Question Answering
- 根据上下文与问题的相关性加权融合多个检索结果。
- 在三个视觉问答基准上均超越现有方法,最高提升4.2%准确率。
- 无需训练,可直接替换解码器,对检索结果不敏感。
尽管大型视觉语言模型(LVLMs)能力强大,但在特定实体的细节知识方面仍显不足。检索增强生成(RAG)通过从外部知识库提供额外上下文来增强LVLM。然而我们发现,以往的RAG解码方法未能充分利用多个相关上下文,且无法有效抑制无关上下文的干扰。为此,本文提出相关性感知的多上下文对比解码(RMCD),通过结合每个上下文生成的输出,并按其与问题的相关性加权,实现信息的有效聚合,同时缓解无关内容的影响。实验表明,RMCD在多个LVLM上持续优于其他解码方法,在三个知识密集型视觉问答基准上表现最佳。该方法可直接替换现有模型的解码器,无需额外训练。分析显示,其性能在弱到强的检索结果下均稳定领先。代码已开源。
原文摘要 · Abstract (English)
Despite the remarkable capabilities of Large Vision Language Models (LVLMs), they still lack detailed knowledge about specific entities. Retrieval-augmented Generation (RAG) is a widely adopted solution that enhances LVLMs by providing additional contexts from an external Knowledge Base. However, we observe that previous decoding methods for RAG are sub-optimal as they fail to sufficiently leverage multiple relevant contexts and suppress the negative effects of irrelevant contexts. To this end, we propose Relevance-aware Multi-context Contrastive Decoding (RMCD), a novel decoding method for RAG. RMCD outputs a final prediction by combining outputs predicted with each context, where each output is weighted based on its relevance to the question. By doing so, RMCD effectively aggregates useful information from multiple relevant contexts while also counteracting the negative effects of irrelevant ones. Experiments show that RMCD consistently outperforms other decoding methods across multiple LVLMs, achieving the best performance on three knowledge-intensive visual question-answering benchmarks. Also, RMCD can be simply applied by replacing the decoding method of LVLMs without additional training. Analyses also show that RMCD is robust to the retrieval results, consistently performing the best across the weakest to the strongest retrieval results. Code is available at https://github.com/mlvlab/RMCD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。