通过自适应语义路由，智能保留关键视觉片段，避免模型因过早丢弃信息而失效。
Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models
- 基于跨模态注意力与熵值动态调整保留策略，不依赖训练
- 在7个基准上减少77.8%视觉标记，速度提升2.15倍，性能损失仅1.36%
- 适合需要高效推理且保持多粒度视觉理解的多模态应用
现有剪枝方法常假设低注意力视觉标记可被丢弃，仅依据浅层图文注意力排序并移除低分块以加速视觉语言模型推理。我们发现，这种标量标准在组合推理中不可靠：早期被忽略的标记可能在后续阶段对识别次级对象、空间关系和上下文线索至关重要。过早剪枝会导致视觉失语症——模型失去视觉依据，转而依赖语言先验。本文提出COAST（COntrastive Adaptive Semantic Token Pruning），一种无需训练的剪枝框架，将压缩视为自适应语义路由。COAST利用原生跨模态注意力识别查询相关锚点，通过注意力熵估计上下文分散性，并动态权衡语义证据与空间上下文保留。进一步引入对比路由分数，同时保留锚点对齐证据与互补空间上下文。在七个基准上，COAST将视觉标记减少77.8%，实现2.15倍延迟加速，平均性能保留98.64%。该方法在不同标记预算下优于强基线，且在多个LVLM家族间具有泛化能力，表明自适应语义路由是单次标量剪枝的稳健替代方案。
原文摘要 · Abstract (English)
Are low-attention visual tokens truly redundant in vision-language reasoning? Existing pruning methods often assume so, ranking visual tokens by shallow text-to-image attention and discarding low-scoring patches to accelerate LVLM inference. We show that this scalar criterion is unreliable for compositional reasoning: tokens ignored in early layers can later become essential for resolving secondary objects, spatial relations, and contextual cues. Premature pruning can therefore induce Visual Aphasia, a failure mode in which the model loses visual grounding and falls back on language priors. We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing. COAST uses native cross-modal attention to identify query-specific anchors and estimate contextual dispersion via attention entropy, then adapts the retention trade-off between semantic evidence and spatial context. It further uses a contrastive routing score to preserve both anchor-aligned evidence and complementary spatial context. Across seven benchmarks, COAST reduces visual tokens by 77.8% and achieves a 2.15x latency speedup while retaining 98.64% of the original average performance. Beyond a single backbone or compression setting, COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families, showing that adaptive semantic routing is a robust alternative to one-shot scalar pruning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。