根据问题动态分配视觉标记预算,提升图文模型推理效率。
ET-Prune: Evidence-Aware Dynamic Budgeting for Visual Token Pruning in Text-Rich MLLMs
- 基于解码器侧查询键块生成问题相关证据,指导剪枝
- 在保留约一半视觉标记下,多项任务表现超越现有剪枝方法
- 适合文本密集型多模态任务,如OCR识别与问答
视觉标记剪枝可降低多模态大模型的推理开销,但固定剪枝比例难以适配文本丰富的输入。在以OCR为核心的任务中,关键证据可能仅为少量文字、标签或字段,其重要性由问题决定;盲目剪枝可能剔除此类证据,而保留视觉显著但无关区域。我们提出ET-Prune,一种无需训练的框架,将剪枝视为证据分配过程。它从解码器侧部分查询-键块中推导出问题相关的证据,保护文本类空间区域,并将证据不确定性与密度转化为样本特异的最小保留标记数。三个渐进式中间层事件逐步将序列推向该预算:对分散或文本密集的证据保留更多标记,对集中证据更激进地剪枝。在每个配置仅一次确定性遍历的观测点估计下,ET-Prune在六个骨干模型-基准组合中均领先或持平于剪枝方法,在约一半视觉标记保留条件下实现最优性能。在OCRBench-v2上,其在Qwen3-VL-8B和InternVL3.5-8B上分别领先最强剪枝基线1.80和0.68个百分点;在MMBench v1.1上,达到0.8467的圆形精确匹配准确率,优于原生模型的0.8437,平均视觉标记保留率为54.45%。结果表明,证据感知的动态预算策略在文本丰富型多模态推理中具有良好的质量-成本权衡。
原文摘要 · Abstract (English)
Visual token pruning reduces the inference cost of multimodal large language models, but a fixed token ratio is poorly matched to text-rich inputs. In OCR-centric tasks, decisive evidence can be a small number, label, or field whose relevance is specified by the question; indiscriminate pruning can erase that evidence while retaining visually salient but irrelevant regions. We present ET-Prune, a training-free framework that casts pruning as evidence allocation. It derives question-conditioned evidence from a decoder-side partial query-key block, safeguards text-like spatial regions, and converts evidence uncertainty and density into a sample-specific token floor. Three progressive middle-layer events then move the sequence toward this budget, retaining more tokens for diffuse or text-dense evidence and pruning concentrated evidence more aggressively. At the observed point estimates from one deterministic pass per configuration, ET-Prune leads or ties among pruned methods in all six backbone-benchmark comparisons at roughly half tokens. On OCRBench-v2, it leads the strongest pruned baselines by 1.80 and 0.68 percentage points on Qwen3-VL-8B and InternVL3.5-8B, respectively, while retaining about half of the visual tokens; on MMBench v1.1, it reaches 0.8467 circular exact-matching accuracy versus 0.8437 for Vanilla at 54.45% average visual-token retention. These results show a favorable observed quality-cost trade-off for evidence-aware dynamic budgeting in text-rich multimodal inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。