自动挑选重要视觉标记，提速模型推理且不损失精度
The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gating
- 用噪声门控机制动态评估每个视觉标记重要性
- 在10个基准上保留96.5%准确率，预填充加速2.85倍
- 无需额外标注，可通用适配不同视觉语言模型
视觉标记在视觉-语言模型（VLMs）中主导计算开销，但其中许多信息冗余。现有剪枝方法通常依赖注意力幅度或相似性评分。本文将视觉标记剪枝重新建模为容量受限的通信问题：给定固定预算K，模型需在有限带宽下最大化保留视觉信息。提出AutoSelect，在冻结的VLM上附加轻量级Scorer与Denoiser，仅使用标准的下一个词预测损失进行训练，无需辅助目标或额外标注。训练时，保持方差的噪声门控根据预测重要性调节每个标记的信息流，使梯度能通过所有标记；对角注意力Denoiser则恢复扰动后的表示。推理时仅保留Scorer和硬性top-K选择，延迟可忽略。在10个VLM基准上，AutoSelect在仅0.69毫秒开销下实现2.85倍的LLM预填充加速，同时保持96.5%的全模型精度，并可无需架构调优迁移至不同VLM主干网络。代码已开源。
原文摘要 · Abstract (English)
Visual tokens dominate inference cost in vision-language models (VLMs), yet many carry redundant information. Existing pruning methods alleviate this but typically rely on attention magnitude or similarity scores. We reformulate visual token pruning as capacity constrained communication: given a fixed budget K, the model must allocate limited bandwidth to maximally preserve visual information. We propose AutoSelect, which attaches a lightweight Scorer and Denoiser to a frozen VLM and trains with only the standard next token prediction loss, without auxiliary objectives or extra annotations. During training, a variance preserving noise gate modulates each token's information flow according to its predicted importance so that gradients propagate through all tokens; a diagonal attention Denoiser then recovers the perturbed representations. At inference, only the Scorer and a hard top-K selection remain, adding negligible latency. On ten VLM benchmarks, AutoSelect retains 96.5% of full model accuracy while accelerating LLM prefill by 2.85x with only 0.69 ms overhead, and transfers to different VLM backbones without architecture-specific tuning. Code is available at https://github.com/MedHK23/AutoSelect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。