DREAM让多模态模型生成更快更准，速度最高提升3.6倍。
DREAM: Drafting with Refined Target Features and Entropy-Adaptive Cross-Attention Fusion for Multimodal Speculative Decoding
- 用交叉注意力融合目标模型中间特征，提升草稿与主模型对齐
- 根据注意力熵自适应选择特征，训练更高效，草稿接受率更高
- 压缩视觉令牌降低延迟，适合高并发多模态生成场景
推测解码（SD）已成为加速大语言模型自回归生成的有效方法，但在视觉-语言模型（VLMs）中的应用仍不充分。本文提出DREAM，一种专为VLM设计的新型推测解码框架，包含三项核心创新：(1) 基于交叉注意力机制，将目标模型的中间特征注入草稿模型以增强对齐；(2) 依据注意力熵自适应选择中间特征，指导草稿模型高效训练；(3) 视觉令牌压缩以降低草稿模型延迟。DREAM实现高效、准确且并行的多模态解码，显著提升吞吐量。在包括LLaVA、Pixtral、SmolVLM和Gemma3在内的多个主流VLM上实验表明，相比传统解码最高提速3.6倍，且在推理吞吐量和推测草稿接受长度上均显著优于现有基线。代码已开源：https://github.com/SAI-Lab-NYU/DREAM.git
原文摘要 · Abstract (English)
Speculative decoding (SD) has emerged as a powerful method for accelerating autoregressive generation in large language models (LLMs), yet its integration into vision-language models (VLMs) remains underexplored. We introduce DREAM, a novel speculative decoding framework tailored for VLMs that combines three key innovations: (1) a cross-attention-based mechanism to inject intermediate features from the target model into the draft model for improved alignment, (2) adaptive intermediate feature selection based on attention entropy to guide efficient draft model training, and (3) visual token compression to reduce draft model latency. DREAM enables efficient, accurate, and parallel multimodal decoding with significant throughput improvement. Experiments across a diverse set of recent popular VLMs, including LLaVA, Pixtral, SmolVLM and Gemma3, demonstrate up to 3.6x speedup over conventional decoding and significantly outperform prior SD baselines in both inference throughput and speculative draft acceptance length across a broad range of multimodal benchmarks. The code is publicly available at: https://github.com/SAI-Lab-NYU/DREAM.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。