测试大模型在反复追问下的稳定性,发现持续质疑会让答案乱翻,不同模型表现差异大。
Just Keep Prompting: Evaluating Repetitive Socratic Prompting in VLMs

- 设计三类追问策略,测试视觉语言模型在10轮对话中的认知稳定性。
- 多数模型在持续挑战下准确率波动大,正确答案会变错,错误答案会反弹。
- 揭示模型在压力下的反应模式,适合关注多轮交互可靠性的研究者。
将视觉语言模型(VLMs)部署于真实场景不仅需要强大的视觉推理能力,还需在持续对话压力下保持稳定。我们提出「只需不断提问」(JKP)框架,评估模型在用户反复质疑或反驳时的信念稳定性。该框架通过三种策略对模型进行最多10轮追问:对抗性否定(反复拒绝)、纯苏格拉底式质询(反复要求重新评估确定性)、上下文感知的苏格拉底总结(回放先前推理再要求重审)。我们在STAR基准子集上对GPT-4o、Gemini 2.5 Pro和Qwen3-VL-30B进行了720次多轮测试。整体准确率从第0轮到第10轮变化不大,但轨迹分析显示显著不稳定性:正确答案会回退,错误答案会恢复,大量实验出现答案反复翻转。重复提问虽有有限增益,但常成为不稳定因素而非推理助力。效果高度依赖模型:Qwen3-VL-30B最终准确率最高,但在直接反驳下变得自信错误;Gemini 2.5 Pro相对稳定但消耗大量令牌;GPT-4o最脆弱且震荡剧烈。这些发现表明,多轮评估不仅捕捉额外推理,还揭示模型在压力下的响应特征——如何在视觉锚定、校准与对话合规之间权衡。
原文摘要 · Abstract (English)
Deploying Vision-Language Models (VLMs) in real-world settings requires not only strong visual reasoning but also stability under sustained conversational pressure. We introduce Just Keep Prompting (JKP), a multi-turn evaluation framework that measures VLM epistemic stability when users repeatedly challenge, question, or contradict a model's answer. JKP probes models for up to 10 follow-up turns using three strategies: Adversarial Negation (repeated rejection), Pure Socratic Interrogation (repeated calls to reassess certainty), and Context-Aware Socratic Summarization (reflecting the model's prior rationale back before asking for reconsideration). We evaluate GPT-4o, Gemini 2.5 Pro, and Qwen3-VL-30B on a subset of the STAR benchmark across 720 multi-turn runs. Aggregate accuracy changes modestly from Turn 0 to Turn 10, but trajectory-level analysis reveals substantial instability: correct answers regress, wrong answers recover, and many runs exhibit repeated answer flipping. Repeated prompting has bounded upside and often acts as a destabilizer rather than a reasoning aid. The effect is strongly model-dependent: Qwen3-VL-30B achieves the highest final accuracy but becomes confidently wrong under direct contradiction; Gemini 2.5 Pro is comparatively stable but token-expensive; GPT-4o is the most brittle and oscillatory. These findings reveal that multi-turn VLM evaluation captures not just additional reasoning but pressure-response profiles: how models trade off visual grounding, calibration, and conversational compliance under repeated challenge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。