用大模型提升视觉问答的实体检索准确率，解决外观相似但概念不同问题。
Beyond Visual Similarity: Entity-Aligned Retrieval for Knowledge-Based Visual Question Answering
- 用大模型生成语义对齐的图像嵌入，避免仅依赖表面视觉相似性
- 在大规模数据上实现14.7%的召回率提升和9.4%的问答准确率增益
- 适合需要精准跨模态实体匹配的研究者与工业应用
基于知识的视觉问答（KB-VQA）依赖外部信息来回答涉及长尾实体的问题。现有检索流程多采用CLIP风格的双编码器，侧重表面视觉相似性，而非实体级语义对齐，导致语义相同但视觉差异大的概念难以匹配，或不同实体因外观相似而混淆。为此，我们提出首个面向KB-VQA的大模型嵌入检索器KBMR。利用大语言模型的自回归能力，将图像映射到更保留概念身份的语义空间。针对维基百科规模检索中的噪声标注问题，引入基于大模型的语义判别器，生成连续的实体一致性权重。这些权重驱动一种新型连续语义蒸馏目标，实现有效的硬负样本采样与超越二值标签的软监督。大量实验表明，KBMR显著优于CLIP基线，在检索召回率@1上提升达14.7%，端到端VQA准确率提升9.4%。代码已公开于https://github.com/realHarryX/KBMR。
原文摘要 · Abstract (English)
Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。