arXiv:2608.22883cs.CV2026-08

动态按需调用视觉证据,让多模态生成更快更准。

FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding

论文配图:FOVEA: Focused On-Demand Visual Evidence Adaptation for Cache-Friendly Multimodal Speculative Decoding
图 1 · 摘自论文原文
  • 根据当前状态动态检索视觉记忆中的关键片段
  • 实现最高2.13倍的解码加速,提升候选词接受率
  • 适合追求高效多模态生成的开发者和研究者

多模态推测解码通过轻量级草稿模型并行提出候选词,由大型目标模型验证以加速视觉语言模型。现有方法通常依赖固定的视觉接口,如预设的视觉令牌预算或静态压缩表示。我们控制性分析表明,不同任务和解码阶段对视觉需求差异显著,过多视觉输入未必有益,反而可能引入冗余或干扰。为此提出FOVEA(聚焦式按需视觉证据自适应),构建可复用的视觉记忆,并动态检索有界子集用于草稿状态。累积质量规则决定选择数量与内容,所选条目聚合为视觉读出,通过轻量门控残差修正与当前草稿隐藏状态融合。该修正仅作用于传递给语言模型头的表示,不插入视觉令牌至自回归上下文。在多个视觉语言骨干网络与多模态基准上实验表明,FOVEA提升了草稿接受率与端到端解码速度,相比自回归解码最高提速2.13倍。结果证明,基于状态的证据检索是替代固定视觉表示的有效方案。

原文摘要 · Abstract (English)

Multimodal speculative decoding accelerates vision-language models by allowing a lightweight draft model to propose candidate tokens for parallel verification by a larger target model. Existing methods typically condition the drafter on a fixed visual interface, such as a predefined visual-token budget or a static compressed representation. However, our controlled visual-budget analysis shows that visual demand varies substantially across tasks and decoding stages, which means more visual input is not always beneficial. Actually, insufficient evidence may weaken visual grounding, while excessive context adds overhead and may disrupt drafting. We propose FOVEA (Focused On-demand Visual Evidence Adaptation), a cache-friendly approach that builds a reusable visual memory and dynamically retrieves a bounded subset for a draft state. A cumulative-mass rule determines both how many and which entries are selected. The selected entries are aggregated into a visual readout and fused with the current draft hidden state through a lightweight gated residual correction. Rather than inserting visual tokens into the autoregressive context, the correction modifies only the representation passed to the language-model head. Experiments across multiple vision-language backbones and multimodal benchmarks show that FOVEA improves draft acceptance and end-to-end decoding speed, achieving up to $2.13\times$ speedup over autoregressive decoding. These results demonstrate that state-conditioned evidence retrieval is an effective alternative to reusing a fixed visual representation throughout multimodal generation.

多模态生成推测解码视觉记忆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。