让视觉语言模型更准地搜索图像，靠的是自我预言式解码。
Self-Prophetic Decoding to Unlock Visual Search in LVLMs
- 用预训练模型做预言，后训练模型选择性采纳其输出。
- 在12个图像搜索任务中全部提升，且不增加计算开销。
- 适合想提升多步推理能力的视觉语言模型研究者。
大型视觉语言模型（LVLMs）正朝着真正的多模态推理演进，视觉搜索是‘以图思考’理念的具体体现。然而，后训练后的LVLM在视觉搜索中面临两大挑战：内在能力不兼容，以及长链多步推理中的干扰。本文提出两个新见解：首先，预训练与后训练模型间的自调节机制，利用预训练模型的单步能力缓解能力退化和长上下文干扰；其次，基于概率的预言采样取代传统提示，使预训练模型充当‘先知’，后训练模型在其输出分布内选择性接受预言符号，保持推理连贯性。基于此，我们提出SeProD——一种无需训练、即插即用的自预言解码框架，通过内在单步能力实现连贯的多步推理。实验表明，SeProD在4个视觉搜索基准的全部12个划分上均一致提升多个视觉搜索LVLM，并在通用VQA基准上表现优异，且因并行预言接受机制无额外计算开销。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) are rapidly evolving toward true multimodal reasoning, with visual search representing a concrete instantiation of the thinking-with-images paradigm. However, LVLM visual search faces two key challenges: incompatibility among intrinsic capabilities after post-training, and interference in long multi-step reasoning contexts. To address these, we identify two novel insights. First, self-regulation between pre- and post-training LVLMs leverages the intrinsic single-step capabilities of the pre-training model to mitigate capability deterioration and long-context interference. Second, probability-based prophetic sampling, replacing naive prompting, provides a probabilistic interface where the pre-training model acts as a prophet and the post-training model selectively accepts prophetic tokens under its output distribution, preserving coherent multi-step reasoning. Building on these insights, we introduce SeProD, a self-prophetic decoding framework that leverages intrinsic single-step capabilities to enable coherent multi-step reasoning in a training-free, plug-and-play manner. Experiments show that SeProD consistently improves multiple visual-search LVLMs across all 12 splits of 4 visual search benchmarks, as well as across general VQA benchmarks, without added computational overhead, thanks to its parallel prophetic acceptance mechanism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。