arXiv:2505.12728cs.CVcs.MM2025-05被引 2

提出SpecFLASH框架,让多模态生成更快更准。

SpecFLASH: A Latent-Guided Semi-autoregressive Speculative Decoding Framework for Efficient Multimodal Generation

  • 用潜空间引导压缩视觉令牌,减少冗余
  • 半自回归预测多个视觉标记,提速最高2.68倍
  • 适合视频字幕、视觉指令调优等多模态任务

大语言模型和大多模态模型(LLMs 和 LMMs)生成效果强但推理慢,尤其在处理视觉输入时,其序列通常包含更多低信息密度的令牌。推测解码通过小型草稿模型预估候选令牌,由主模型选择性采纳,可在不降低质量的前提下加速推理。然而现有方法大多忽略视觉表征的结构特性,且依赖纯文本草稿模型。本文提出专为LMM设计的SpecFLASH框架,显式利用多模态结构优化草稿模型。首先通过轻量级潜空间引导的令牌压缩模块,在保留语义的同时压缩视觉特征;再通过半自回归解码机制,利用视觉实体的共现与局部相关性,单次前向传播预测多个令牌。大量实验表明,SpecFLASH持续优于现有推测解码基线,在视频字幕任务上最高达2.68倍加速,在视觉指令调优任务上达2.55倍加速,显著超越原LMM。代码已开源:https://github.com/ZihuaEvan/FlashSD/

原文摘要 · Abstract (English)

Large language models and large multimodal models (LLMs and LMMs) deliver strong generative performance but suffer from slow decoding, a problem that becomes more severe when handling visual inputs, whose sequences typically contain many more tokens with lower information density than text. Speculative decoding accelerates LLM inference by letting a compact draft model propose candidate tokens that are selectively accepted by a larger target model, achieving speed-up without degrading quality. However, existing multimodal speculative decoding approaches largely ignore the structural characteristics of visual representations and usually rely on text-only draft models. In this paper, we introduce SpecFLASH, a speculative decoding framework tailored to LMMs that explicitly exploits multimodal structure when designing the draft model. We first mitigate redundancy in visual token sequences with a lightweight, latent-guided token compression module that compacts visual features while preserving semantics, and then leverage the co-occurrence and local correlations of visual entities via a semi-autoregressive decoding scheme that predicts multiple tokens in a single forward pass. Extensive experiments demonstrate that SpecFLASH consistently surpasses prior speculative decoding baselines, achieving up to $2.68\times$ speed-up on video captioning and $2.55\times$ on visual instruction tuning, relative to the original LMM. Our code is available here: https://github.com/ZihuaEvan/FlashSD/.

多模态生成推测解码视觉压缩半自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。