arXiv:2601.20168cs.CV2026-01被引 1

针对视觉语言模型的计算冗余问题,提出分层剪枝方法提升效率。

Efficient Token Pruning for LLaDA-V

  • 在中间到晚期层选择性删除视觉标记,减少重复计算。
  • 最高降低65%计算量,平均保持95%任务性能。
  • 首次为扩散模型设计视觉感知剪枝,适合高效推理场景。

基于扩散的大型多模态模型(如 LLaDA-V)在视觉-语言理解与生成方面表现出色,但其双向注意力机制和扩散式迭代去噪范式带来了显著计算开销,因视觉标记需在所有层和去噪步骤中反复处理。本文通过深入分析注意力模式发现,与自回归解码器不同,LLaDA-V 的跨模态信息主要在中后期层聚合,导致语义对齐延迟。受此启发,我们提出受 FastV 启发的结构化标记剪枝策略,仅在特定层有选择地移除部分视觉标记,以降低浮点运算量(FLOPs)同时保留关键语义信息。据我们所知,这是首个研究扩散型多模态模型结构化剪枝的工作。不同于 FastV 专注浅层剪枝,本方法聚焦于第一轮去噪步骤的中后期层,契合 LLaDA-V 的延迟注意力聚合特性,从而维持输出质量;且首轮剪枝可减少后续所有步骤的计算负担。在多个基准测试中,最佳配置将计算成本降低高达65%,平均保持95%的任务性能,为高效推理提供实证基础,并凸显视觉感知剪枝在扩散型多模态模型中的潜力。

原文摘要 · Abstract (English)

Diffusion-based large multimodal models, such as LLaDA-V, have demonstrated impressive capabilities in vision-language understanding and generation. However, their bidirectional attention mechanism and diffusion-style iterative denoising paradigm introduce significant computational overhead, as visual tokens are repeatedly processed across all layers and denoising steps. In this work, we conduct an in-depth attention analysis and reveal that, unlike autoregressive decoders, LLaDA-V aggregates cross-modal information predominantly in middle-to-late layers, leading to delayed semantic alignment. Motivated by this observation, we propose a structured token pruning strategy inspired by FastV, selectively removing a proportion of visual tokens at designated layers to reduce FLOPs while preserving critical semantic information. To the best of our knowledge, this is the first work to investigate structured token pruning in diffusion-based large multimodal models. Unlike FastV, which focuses on shallow-layer pruning, our method targets the middle-to-late layers of the first denoising step to align with LLaDA-V's delayed attention aggregation to maintain output quality, and the first-step pruning strategy reduces the computation across all subsequent steps. Our framework provides an empirical basis for efficient LLaDA-V inference and highlights the potential of vision-aware pruning in diffusion-based multimodal models. Across multiple benchmarks, our best configuration reduces computational cost by up to 65% while preserving an average of 95% task performance.

多模态模型剪枝技术扩散模型高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。