arXiv:2604.12035cs.CV2026-04被引 1

视觉标记剪枝不仅影响效率,还显著改变模型置信度可靠性。

When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMs

论文配图:When Does Visual Token Pruning Improve Calibration? The Role of Evidence Coverage in MLLMs
图 1 · 摘自论文原文
  • 基于证据覆盖率选择保留的视觉标记,能更好保持校准性。
  • 从576减至128个标记时,校准误差从0.041降至0.016。
  • 适合关注模型置信度质量的研究者与部署优化者。

视觉标记剪枝广泛用于降低多模态大语言模型(MLLMs)的推理成本,但通常仅以准确率评估。本文研究剪枝对校准性的影响,即置信度与正确性的一致性,发现选择策略比剪枝比例更重要。在POPE数据集上,将LLaVA-1.5的视觉标记从576个降至128个,基于覆盖率的剪枝使期望校准误差从0.041降至0.016,且准确率无显著下降。而基于注意力的选择虽维持置信度,但准确率下降,导致校准性劣于随机剪枝。在多种剪枝条件下,保留集覆盖率与准确率强相关(Spearman ρ=+0.89),但与平均置信度无关(ρ=-0.03),与过度自信呈强负相关(ρ=-0.92)。可控干预实验支持该证据覆盖率解释。两个边界限制其适用:在查询条件化的FastV中,其过度自信超出覆盖率预测;在语言主导的ScienceQA上,覆盖率无法排序校准性。该方法在GQA和LLaVA-NeXT中具泛化性,在Qwen2-VL上也优于随机剪枝,但校准提升取决于任务与模型。此外,识别出FastV中的评估陷阱:将剪枝标记设为零而非移除会令准确率降为随机水平。因此,剪枝不仅影响效率,也改变置信度质量,评估时应同步考察校准性。

原文摘要 · Abstract (English)

Visual token pruning is widely used to reduce the inference cost of multimodal large language models (MLLMs), but it is usually evaluated only by accuracy. We study how pruning affects calibration, defined as the agreement between confidence and correctness, and show that the selection rule matters more than the token budget alone. On POPE with LLaVA-1.5, coverage-based pruning from 576 to 128 tokens reduces expected calibration error from 0.041 to 0.016 without a statistically significant accuracy loss. In contrast, attention-based selection preserves confidence while accuracy deteriorates, becoming less calibrated than random pruning at aggressive budgets. Across pruning conditions, kept-set coverage is strongly associated with accuracy (Spearman $ρ=+0.89$) but not with mean confidence ($ρ=-0.03$), producing a strong inverse relation with overconfidence ($ρ=-0.92$); controlled kept-set interventions support this evidence-coverage account. Two boundaries limit it: query-conditioned FastV is more overconfident than its coverage predicts, and on language-prior-dominated ScienceQA, coverage ceases to order calibration. The selector ordering otherwise generalizes to GQA and LLaVA-NeXT, and coverage beats random on Qwen2-VL, although the calibration gain over the unpruned model is task- and model-dependent. We also identify an evaluation pitfall in FastV: zeroing rather than removing pruned tokens can reduce accuracy to chance. Visual token pruning therefore changes confidence quality as well as efficiency, and calibration should be evaluated alongside accuracy when comparing pruning methods.

视觉剪枝校准性多模态模型置信度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。