arXiv:2608.07088cs.CVcs.AI2026-08

让视觉令牌剪枝更智能:按角色分区保留关键区域,提速降耗。

RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs

论文配图:RoRA: Role-Oriented Regional Allocation for Visual Token Pruning in MLLMs
图 1 · 摘自论文原文
  • 基于角色分区的剪枝框架,分核心、上下文和细节三区处理视觉令牌。
  • 在88.9%剪枝率下仍保持96.5%性能,比现有方法快5%以上。
  • 无需训练,仅0.7毫秒选令牌,推理速度提升1.33倍。

多模态大语言模型将图像编码为长视觉令牌序列,导致预填充和键值缓存存储开销大。现有无训练剪枝方法按重要性、多样性或空间覆盖选择令牌,但将保留令牌视为可互换,未显式追踪已覆盖的对象区域。我们提出RoRA,一种无训练框架,将视觉令牌剪枝建模为面向角色的区域证据分配。在固定预算下,RoRA将令牌分为受保护的语义核心、互补上下文与细粒度细节三部分。首先通过位置先验和提示校准的对象先验校准文本条件注意力,再从高置信度锚点构建注意力锚定区域(AARs),作为已覆盖对象支持的轻量代理。上下文主要在AAR外探索,小预算的AAR引导恢复局部细节;仅用成对相似性进行上下文阶段冗余过滤。在匹配预算下,RoRA在LLaVA和Qwen-VL系列上持续优于强基线,在激进剪枝率下仍保持近完整性能,如在LLaVA-1.5上达88.9%剪枝率时保留96.5%原性能,且在Qwen3-VL上75%-90%剪枝率下比D2Pruner提升约5%。在66.7%剪枝率下,令牌选择仅需0.7毫秒,端到端推理时间减少24.6%,相比未剪枝推理实现1.33倍加速(NVIDIA H800)。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) encode images as long visual token sequences, making prefilling and KV-cache storage expensive. Existing training-free pruning methods select tokens by importance, diversity, or spatial coverage, but treat retained tokens as interchangeable and do not explicitly track which object-related regions are already covered. We present RoRA, a training-free framework that casts visual token pruning as role-oriented regional evidence allocation. Given a fixed budget, RoRA partitions tokens into a protected semantic core, complementary context, and fine-grained detail. It first calibrates text-conditioned attention with a positional prior and a prompt-calibrated object prior, then builds Attention-Anchored Regions (AARs) from high-confidence anchors as lightweight proxies for covered object support. Context is explored mainly outside AARs, while a small AAR-guided budget restores local detail; pairwise similarity is used only for context-stage redundancy filtering. Under matched budgets, RoRA consistently outperforms strong training-free baselines across LLaVA and Qwen-VL families, retaining most of the unpruned accuracy even at aggressive pruning ratios, e.g., 96.5% of full performance at 88.9% pruning on LLaVA-1.5, and improving over D2Pruner by about 5% on Qwen3-VL at 75-90% pruning. At a 66.7% pruning ratio, RoRA requires only 0.7 ms for token selection and reduces end-to-end inference time by 24.6%, corresponding to a 1.33x speedup over unpruned inference on an NVIDIA H800.

视觉剪枝多模态推理加速无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。