多模态问答中，结尾信息易被忽略，首段信息更受重视。
Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering
- 通过控制提示位置设计实验，验证多模态问答存在首因效应。
- 首段金句比末段高16至26分，末端信息严重丢失。
- 适合研究检索增强型视觉语言模型的阅读器优化者。
基于知识的视觉问答（KB-VQA）通过从维基规模知识库中检索文本片段，使视觉语言系统回答超出自身参数化知识的问题。在纯文本长上下文大模型中，检索内容的使用呈现典型的U形‘中间迷失’现象：开头和结尾信息被利用，中间内容被忽略。这一现象是否适用于实际部署的多模态KB-VQA尚不明确。为此，我们设计首个针对多模态KB-VQA阅读器位置依赖性的可控探测：金句位置协议，仅改变问题中金句片段的提示位置。我们在三个7B/8B开源视觉语言模型阅读器和两个KB-VQA基准上，将检索数量k扩展至20，发现效果反转为‘首因效应’：首段金句表现优于末段金句16至26分，我们称之为‘末端迷失’。三项消融实验表明：纯文本条件下已存在首因效应，多模态环境使其放大2.2至4.5倍；图像位置与干扰项打乱实验共同定位问题于指令微调阅读器的提示槽0位。在冻结阅读器情况下，三种检索侧改进（MMR、最优重排序、秩重排）均无法缩小差距。研究指出，召回率@k不适合作为部署级KB-VQA的评价指标，修复该差距需阅读器侧干预；我们已发布该协议作为评估干预措施的标准化工具。
原文摘要 · Abstract (English)
Knowledge-based visual question answering (KB-VQA) lets vision-language systems answer questions that exceed their parametric knowledge by conditioning a reader on passages retrieved from a Wikipedia-scale knowledge base. In pure-text long-context LLMs, retrieved-context use follows the U-shaped "lost-in-the-middle" effect of Liu et al. (2024): information at the start and end of context is used, the middle is lost. Whether this transfers to deployed multimodal KB-VQA is open. To close this gap, we design the first controlled probe of reader-side position dependence in multimodal KB-VQA: a gold-position protocol in which only the gold passage's prompt slot varies within question. We run it on three open-source 7B/8B VLM readers and two KB-VQA benchmarks at k up to 20. The shape flips from U to primacy: gold-at-first beats gold-at-last by 16 to 26 points on every reader-by-benchmark cell, an effect we call "Lost at the End". Three targeted ablations narrow the cause: a text-only control shows the multimodal setting amplifies an already-present text-mode primacy 2.2 to 4.5 times, and image-position and distractor-shuffle ablations together pin the locus to prompt slot 0 of the instruction-tuned reader. On a frozen reader, three retrieval-side fixes (MMR, oracle reranking, rank-based reordering) all leave the gap intact (no separable improvement). Our findings indicate that recall@k is the wrong metric for deployed KB-VQA and that closing the gap requires reader-side intervention; we release our protocol as a controlled instrument for evaluating such interventions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。