通过重复提问,让模型既看懂问题又答对题。
Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models

- 在图像前后重复提问,前半句引导关注,后半句便于回答时引用。
- 在多个数据集上提升准确率,最高达19分,且无需训练或改模型。
- 揭示了提问顺序的深层矛盾,适合所有用prompt的视觉语言模型研究者。
视觉语言模型中,问题应放在图像前还是后?直觉认为应放前面:知道问题才能聚焦。但实测发现,问题前置反而表现更差,这一现象称为“问题优先悖论”。我们发现,虽然前置问题确实能引导模型关注相关图像区域,但后续的答题阶段却因被数百个图像特征遮挡,难以读取问题,导致错误答案。因果注意力实验确认:只有当问题在图像之后,答案才能有效访问它。为此提出无需训练的解决方案:在图像前后都重复问题,一前一后分工——前半句引导感知,后半句供答案引用。该策略在五个开源模型上均有效,最大提升达17.5个组准确率点;在NaturalBench、POPE、Winoground和VQAv2等任务上,最高超越单次顺序19个点。该方法模仿人类对“附加问题”的理解方式,同时补充图像整体信息,恢复被遗忘的上下文。其本质是解决“引导感知”与“保留问题可读性”之间的权衡。
原文摘要 · Abstract (English)
Where should the question go in a vision-language model (VLM) prompt: before the image or after it? Intuition says before: knowing what is asked should tell the model where to look. Yet across visual question answering benchmarks, question-first prompting consistently underperforms the image-first ordering recommended for frontier VLMs, a phenomenon we term the question-first paradox. We trace the paradox to a conflict between two stages of VLM computation. Logit-lens and attention probes show the intuition is half right: a question placed before the image genuinely steers perception, moving image patch representations toward question-relevant concepts. The failure lies downstream. Stranded behind hundreds of image tokens, the question is barely attended by the answer token, which instead commits to image-driven (often wrong) answers; a causal attention knockout confirms that the answer reads the question only when the question follows the image. The diagnosis yields a training-free fix: question echoing, restating the question on both sides of the image so that one copy steers perception while the other is read out at answer time. The same division of labor appears in a fifty-year-old finding on human ``adjunct questions'', where repeating a question before and after a passage aids comprehension more than either position alone. Echoing the image as well brings further gains, restoring the whole-image view a causal decoder otherwise loses. The paradox holds across five open VLMs, costing up to 17.5 group-accuracy points. Echoed prompts close it and surpass the best single-pass ordering on NaturalBench, POPE, Winoground, and open-ended VQAv2, by up to 19 Winoground group-accuracy points, with no training, fine-tuning, or architecture change. The paradox reveals a trade-off between steering perception and preserving question access; echoing resolves it through prompt design alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。