arXiv:2507.02909cs.LG2025-07被引 1

通过操作剪枝实现细粒度令牌分配,让关键模块处理更多令牌,提升多模态大模型效率。

Fine-grained Token Allocation Via Operation Pruning for Efficient MLLMs

  • 将计算单元定义为模块处理令牌组的‘操作’,按深度逐层剪枝冗余操作
  • 在仅损失1%性能下,实现86%算力减少和83%延迟降低
  • 适合追求高效推理的多模态大模型部署与边缘计算场景

令牌压缩通过减少冗余令牌加速多模态大语言模型(MLLMs),但忽略了不同模块间结构冗余的差异,导致关键与冗余模块承受相同令牌负载。为此,本文定义‘操作’为模块处理一组令牌的计算单元,提出操作剪枝框架,使模块可选择性处理令牌。基于此,提出深度操作剪枝(DOP),一种数据驱动方法:在小规模验证集上最小化原模型输出概率分布的偏差,同时满足计算约束,搜索最优剪枝策略,将计算预算转移给关键模块以处理更多令牌。DOP采用分层剪枝降低策略空间,并使用加性近似法,将联合偏差分解为各参数独立变化的偏差之和,使验证次数从指数级降至线性。全面评估显示,DOP在6个MLLM和13个基准上超越12个基线,达到新SOTA。在LLaVA-Next-7B上,实测实现86% TFLOPS减少和83%延迟降低,仅损失1%性能。消融实验进一步证明其数据与时间效率及强泛化能力。

原文摘要 · Abstract (English)

Token reduction accelerates Multimodal Large Language Models (MLLMs) by reducing excessive tokens, but overlooks structural redundancy differences, where critical and redundant modules process identical token loads. For fine-grained computation control, we define an ``operation" as the computation for a module to process a group of tokens and introduce the operation pruning framework to enable modules to selectively process tokens. Built on this framework, we propose Depth-wise Operation Pruning (DOP), a data-driven method that searches for strategies to prune redundant operations and save computational budget for critical modules to process more tokens than uniform allocation by minimizing divergence from the original model's output probability distribution on a small validation set while satisfying computational constraints. For efficient optimization, DOP applies depth-wise pruning to reduce policy space and uses an additive approximation to minimize required validation runs. Depth-wise pruning partitions operations by module type and token group, and prunes operations in deeper layers before those in shallower layers within each module-group pair. The additive approximation obtains individual divergences by independently varying each policy parameter, and then sums them to approximate the joint divergence of simultaneously changing all policy parameters, reducing required validation runs from exponential to linear with respect to the number of policy parameters. Comprehensive evaluations show that DOP establishes new state-of-the-art performance across 6 MLLMs and 13 benchmarks against 12 baselines. On LLaVA-Next-7B, DOP achieves 86\% TFLOPS reduction and 83\% latency reduction on real GPU with only 1\% performance loss. Our extensive ablation studies further demonstrate DOP's data and time efficiency as well as strong generalization capabilities.

多模态模型计算优化令牌分配剪枝

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。