arXiv:2608.03450cs.MMcs.AI2026-08中稿 · ACM MM 2026

不训练即可动态切换视觉与逻辑推理模式,提升多模态模型效率与准确率。

Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs

论文配图:Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs
图 1 · 摘自论文原文
  • 用视觉-文本注意力比判断认知焦点,决定何时用隐空间推理
  • 在多个数据集上实现最高准确率,推理步数减少30%以上
  • 适合追求高效推理的多模态应用开发者

多模态大模型推理需兼顾精细视觉感知与严谨逻辑推演。传统显式文本链式思维(CoT)计算开销大且易产生视觉幻觉,现有隐式推理方法通常需昂贵训练。直接将无训练的语言模型推理机制迁移至多模态场景,性能不稳定。我们发现其根源在于依赖词元级熵,混淆了感知模糊性(如视觉细节不清)与逻辑不确定性(如复杂推理步骤)。为此,提出一种无需训练的推理策略,显式解耦感知与推理。设计新度量——视觉-文本注意力比,动态评估模型认知焦点。基于该度量,提出的注意力引导切换(AGS)框架,对感知类词元启用隐式推理以保留连续空间中的高保真视觉信息,对逻辑类词元强制显式文本生成以维持结构锚定。大量实验表明,该方法达到当前最优性能,显著提升准确率并降低自回归步数与延迟。代码已开源:https://github.com/swordAndSnow/MM26-AGS。

原文摘要 · Abstract (English)

Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction. Explicit text-based Chain-of-Thought (CoT) is computationally expensive and prone to visual hallucinations, while existing latent reasoning methods typically require costly training. Furthermore, directly adapting training-free LLM reasoning mechanisms to the multimodal setting yields unstable performance. We identify that this failure stems from their reliance on token-level entropy, which fundamentally conflates perceptual ambiguity (e.g., unclear visual details) with logical uncertainty (e.g., complex reasoning steps). To overcome this bottleneck, we present a novel training-free inference strategy for MLLMs that explicitly decouples perception and reasoning. We propose a novel metric, the vision-to-text attention ratio, to dynamically gauge the model's cognitive focus. Guided by this metric, our proposed framework, Attention-Guided Switching (AGS), adaptively triggers latent reasoning for perceptual tokens to preserve high-fidelity visual information in the continuous space, while enforcing explicit text generation for logical tokens to maintain structural anchoring. Extensive experiments demonstrate that our method achieves state-of-the-art performance, significantly improving both accuracy and inference efficiency by reducing autoregressive steps and latency. Code is released at https://github.com/swordAndSnow/MM26-AGS.

多模态推理无训练推理注意力机制效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。