让小模型也能高效猜大模型的视觉语言输出，提速近50%。
MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models
- 用轻量投影连接视觉编码器，让小模型学看图说话
- 自蒸馏训练使小模型预测与大模型对齐，最多多过30个词
- 适配现有和未来视觉语言模型，适合追求推理加速的开发者
推测解码能通过轻量级草稿模型预先生成多个词元，由大型目标模型并行验证，显著提升语言模型推理速度。但将其应用于视觉语言模型（VLM）面临两大挑战：小型语言模型缺乏处理视觉输入的结构组件，且其词元预测无法匹配考虑视觉上下文的目标VLM。本文提出MASSV，通过两阶段方法将现有小型语言模型转变为有效的多模态草稿模型：首先使用轻量可训练投影器将目标VLM的视觉编码器接入草稿模型；随后利用目标VLM生成的响应进行自蒸馏视觉指令微调，以对齐词元预测。在Qwen2.5-VL和Gemma3模型族上的全面实验表明，MASSV使接受长度提升最高达30%，在视觉引导任务中实现高达1.46倍的端到端推理加速。该方法具有可扩展性与架构兼容性，适用于当前及未来的VLM加速。
原文摘要 · Abstract (English)
Speculative decoding significantly accelerates language model inference by enabling a lightweight draft model to propose multiple tokens that a larger target model verifies simultaneously. However, applying this technique to vision-language models (VLMs) presents two fundamental challenges: small language models that could serve as efficient drafters lack the architectural components to process visual inputs, and their token predictions fail to match those of VLM target models that consider visual context. We introduce Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models (MASSV), which transforms existing small language models into effective multimodal drafters through a two-phase approach. MASSV first connects the target VLM's vision encoder to the draft model via a lightweight trainable projector, then applies self-distilled visual instruction tuning using responses generated by the target VLM to align token predictions. Comprehensive experiments across the Qwen2.5-VL and Gemma3 model families demonstrate that MASSV increases accepted length by up to 30% and delivers end-to-end inference speedups of up to 1.46x on visually-grounded tasks. MASSV provides a scalable, architecture-compatible method for accelerating both current and future VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。