让视觉语言模型像人一样有预见性地聚焦关键区域,减少错误推理。
CoFFT: Chain of Foresight-Focus Thought for Visual Language Models
- 通过三阶段循环模拟人类视觉认知,逐步优化关注点与推理路径。
- 在多个基准上提升3.1%至5.8%,计算开销可控。
- 无需训练,适用于Qwen2.5-VL、InternVL-2.5等主流视觉语言模型。
尽管视觉语言模型(VLMs)取得了显著进展,但其仍受限于视觉输入的复杂性和冗余性。当图像包含大量无关信息时,VLMs 易受干扰,产生过多任务无关的推理过程甚至幻觉。这一局限源于其无法在推理过程中精确发现并处理所需区域。为此,我们提出链式预见-聚焦思维(CoFFT),一种无需训练的新方法,通过模仿人类视觉认知增强VLMs的视觉推理能力。每个预见-聚焦思维包含三个阶段:(1) 多样化样本生成:生成多样化推理样本以探索潜在推理路径,每个样本包含多个推理步骤;(2) 双重预见解码:基于视觉聚焦和推理进展双重标准严格评估这些样本,并将最优样本的第一步加入推理过程;(3) 视觉聚焦调整:在返回第一阶段前,精确调整视觉关注区域以利于后续推理。上述阶段迭代运行,形成推理引导视觉聚焦、视觉聚焦反哺推理的闭环。在Qwen2.5-VL、InternVL-2.5和Llava-Next等多个基准上的实验证明,该方法实现了3.1%至5.8%的一致性能提升,且计算开销可控制。
原文摘要 · Abstract (English)
Despite significant advances in Vision Language Models (VLMs), they remain constrained by the complexity and redundancy of visual input. When images contain large amounts of irrelevant information, VLMs are susceptible to interference, thus generating excessive task-irrelevant reasoning processes or even hallucinations. This limitation stems from their inability to discover and process the required regions during reasoning precisely. To address this limitation, we present the Chain of Foresight-Focus Thought (CoFFT), a novel training-free approach that enhances VLMs' visual reasoning by emulating human visual cognition. Each Foresight-Focus Thought consists of three stages: (1) Diverse Sample Generation: generates diverse reasoning samples to explore potential reasoning paths, where each sample contains several reasoning steps; (2) Dual Foresight Decoding: rigorously evaluates these samples based on both visual focus and reasoning progression, adding the first step of optimal sample to the reasoning process; (3) Visual Focus Adjustment: precisely adjust visual focus toward regions most beneficial for future reasoning, before returning to stage (1) to generate subsequent reasoning samples until reaching the final answer. These stages function iteratively, creating an interdependent cycle where reasoning guides visual focus and visual focus informs subsequent reasoning. Empirical results across multiple benchmarks using Qwen2.5-VL, InternVL-2.5, and Llava-Next demonstrate consistent performance improvements of 3.1-5.8% with controllable increasing computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。