让模型自判何时该画图,提升推理准确率并减少计算开销。
Model Guides You How to Draw: Adaptive Visual Gating for Unified Multimodal Reasoning

- 基于模型内部信号动态判断是否生成视觉步骤。
- 最高提升5.7%准确率,降低91%视觉计算量。
- 无需训练,适合追求高效精准多模态推理的场景。
统一多模态模型(UMMs)在交错推理中生成文本与视觉中间步骤,展现出解决视觉数学推理任务的巨大潜力。然而,我们发现:生成中间视觉步骤并非总是有益,甚至可能因自生成错误视觉证据而误导后续推理。频繁触发视觉步骤还会带来显著的计算与内存开销,降低推理效率。本文观察到,模型内部信号可在视觉生成完成前预判视觉步骤是否有助于推理。具体提出两个信号:1)生成意图,反映模型是否有明确的绘图计划;2)视觉保真度,衡量视觉生成是否保持与原始图像一致。基于此,我们提出AdaViG——一种无需训练的自适应视觉门控方法。AdaViG在早期阶段动态评估每个视觉步骤,当两个信号均弱时即中止生成,从而防止误导性视觉证据进入推理链,同时避免无效计算。大量实验表明,AdaViG可使准确率最高提升5.7%,视觉生成FLOPs降低25.0%-91.0%,壁钟延迟减少15.4%-45.6%。
原文摘要 · Abstract (English)
Unified multimodal models (UMMs) with interleaved reasoning, which generate both textual and visual steps as part of intermediate reasoning traces, have demonstrated great potential for visual mathematical reasoning tasks. However, we identify a key insight in this paradigm: generating intermediate visual reasoning steps is not always beneficial and can even be harmful, as self-generated visual steps may introduce erroneous visual evidence that misleads subsequent reasoning. Moreover, frequently triggering visual steps during reasoning incurs substantial computational and memory overhead, degrading inference efficiency. To address these accuracy and efficiency challenges, we observe that the model's internal signals can indicate whether a visual step will benefit reasoning before the entire visual generation is completed. Specifically, this work identifies two internal signals: 1) Generation Intent, which reflects whether the model has a concrete textual plan for what to draw, and 2) Visual Fidelity, which measures whether the visual generation remains grounded in the original input image. Leveraging these internal signals, we propose AdaViG, a training-free adaptive visual gating method for unified multimodal reasoning. AdaViG dynamically evaluates each triggered visual step at an early visual generation stage and aborts it when both signals are weak, thereby preventing misleading visual evidence from entering the reasoning trace while avoiding unnecessary computation. Comprehensive experiments demonstrate that AdaViG improves accuracy by up to 5.7% while reducing visual generation FLOPs by 25.0%-91.0% and wall-clock latency by 15.4%-45.6%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。