让视觉语言模型更快推理，通过隐藏视觉标记提升生成效率
HiViS: Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models
- 用目标模型做语义融合，让草稿模型间接获取视觉信息
- 平均接受长度提升，推理速度最高加速3.2倍
- 适合追求高效视觉生成的开发者和研究者
推测解码在大语言模型中已证明能有效加速推理，但其扩展至视觉语言模型（VLMs）受限于视觉标记带来的计算负担与语义不一致。近期研究发现，大型VLM中的视觉标记高度冗余，多数可移除而不影响生成质量。受此启发，我们提出HiViS（Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models），利用目标VLM作为语义融合模型，使草稿模型无需显式处理视觉标记即可获得视觉信息，确保草稿阶段的前缀长度与文本标记一致。此外，HiViS采用时间步感知的对齐训练方案，使草稿模型在独立起草过程中能自主传播并优化指令性视觉-文本语义，由依赖步骤的偏差修正残差引导。在多个代表性VLM与基准测试上的大量实验表明，HiViS在平均接受长度和加速比上均实现显著提升。
原文摘要 · Abstract (English)
Speculative decoding has proven effective for accelerating inference in Large Language Models (LLMs), yet its extension to Vision-Language Models (VLMs) remains limited by the computational burden and semantic inconsistency introduced by visual tokens. Recent studies reveal that visual tokens in large VLMs are highly redundant, and most of them can be removed without compromising generation quality. Motivated by this observation, we propose HiViS (Hiding Visual Tokens from the Drafter for Speculative Decoding in Vision-Language Models), a framework that utilizes the target VLM as a semantic fusion model, allowing the drafter to obtain visual information without explicitly processing visual tokens, ensuring that the drafter's prefill sequence length matches that of the textual tokens. Furthermore, HiViS employs a time-step-aware aligned training scheme that allows the drafter to autonomously propagate and refine instructive visual-textual semantics during independent drafting, guided by step-dependent bias-correction residuals. Extensive experiments across representative VLMs and benchmarks demonstrate that HiViS achieves significant improvements in average acceptance length and speedup ratio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。