DREAM-S让视觉语言模型生成提速近4倍，自动匹配最佳推理策略。
DREAM-S: Speculative Decoding with Searchable Drafting and Target-Aware Refinement for Multimodal Generation
- 用搜索式架构自动选最优生成策略和模型结构
- 实测速度比传统方法快3.85倍，优于现有加速方案
- 适合追求高效多模态生成的开发者与研究者
推测解码（SD）已被证明是加速大语言模型自回归生成的有效技术，但其在视觉语言模型（VLMs）中的应用仍较少。我们提出一种名为 DREAM-S 的新型 SD 框架，专为 VLM 的快速高效解码设计。DREAM-S 利用神经架构搜索（NAS）框架，结合目标感知超网络训练，自动识别草案模型与目标模型之间的最优交互策略，以及适配底层硬件平台的最佳草案模型架构。此外，DREAM-S 引入基于注意力熵引导的自适应中间特征蒸馏，以实现高效的草案训练。在多个主流 VLM 上的实验表明，DREAM-S 相较于标准解码方法实现了高达 3.85 倍的速度提升，并显著优于现有的 SD 基线。代码已公开：https://github.com/SAI-Lab-NYU/DREAM-S。
原文摘要 · Abstract (English)
Speculative decoding (SD) has proven to be an effective technique for accelerating autoregressive generation in large language models (LLMs) however, its application to vision-language models (VLMs) remains relatively unexplored. We propose~\textit{DREAM-S}, a novel SD framework designed specifically for fast and efficient decoding in VLMs. DREAM-S leverages a neural architecture search (NAS) framework with target-aware supernet training to automatically identify both the optimal interaction strategy between the draft and target models, and the most suitable draft model architecture for the underlying hardware implementation platform. DREAM-S additionally incorporates adaptive intermediate feature distillation, guided by attention entropy, to enable efficient draft training. Experiments on a range of well-established VLMs show that DREAM-S achieves up to a $3.85\times$ speedup compared to standard decoding approaches and significantly outperforms existing SD baselines. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM-S .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。