无需训练即可加速扩散视觉语言模型推理，通过遮蔽响应词判断视觉标记重要性。
RedVTP: Training-Free Acceleration of Diffusion Vision-Language Models Inference via Masked Token-Guided Visual Token Pruning
- 基于掩码响应词注意力估计视觉标记重要性，动态剪枝不关键标记。
- 在LLaDA-V上提升吞吐量186%，在LaViDa上降低延迟64.97%。
- 适用于追求高效推理的扩散式多模态模型部署，无需重新训练。
视觉语言模型（VLMs）在多模态推理与生成中取得显著进展，但其高计算需求仍是主要挑战。扩散视觉语言模型（DVLMs）因其支持并行标记解码而备受关注，但大量视觉标记仍严重制约推理效率。尽管自回归式VLMs（AVLMs）已有广泛视觉标记剪枝研究，但针对DVLMs的研究仍几乎空白。本文提出RedVTP，一种响应驱动的视觉标记剪枝策略，利用DVLM的推理动态特性。该方法通过掩码响应标记的注意力估算视觉标记重要性。基于重要性分数在各步骤间保持一致的观察，RedVTP在首次推理后剪除低重要性视觉标记，最大化推理效率。实验表明，RedVTP使LLaDA-V和LaViDa的标记生成吞吐量分别提升186%和28.05%，推理延迟分别降低64.97%和21.87%，且未损害——甚至在某些情况下提升了——准确性。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have achieved remarkable progress in multimodal reasoning and generation, yet their high computational demands remain a major challenge. Diffusion Vision-Language Models (DVLMs) are particularly attractive because they enable parallel token decoding, but the large number of visual tokens still significantly hinders their inference efficiency. While visual token pruning has been extensively studied for autoregressive VLMs (AVLMs), it remains largely unexplored for DVLMs. In this work, we propose RedVTP, a response-driven visual token pruning strategy that leverages the inference dynamics of DVLMs. Our method estimates visual token importance using attention from the masked response tokens. Based on the observation that these importance scores remain consistent across steps, RedVTP prunes the less important visual tokens from the masked tokens after the first inference step, thereby maximizing inference efficiency. Experiments show that RedVTP improves token generation throughput of LLaDA-V and LaViDa by up to 186% and 28.05%, respectively, and reduces inference latency by up to 64.97% and 21.87%, without compromising-and in some cases improving-accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。