动态迭代筛选视觉关键信息，大幅减少计算量仍保持高精度
DIVE: Dynamic Iterative Visual Evidence Construction for Efficient Vision-Language Models
- 逐轮选择最相关视觉片段，边保留边更新剩余证据
- 视觉令牌减少88.9%时，性能仍保持原模型98.2%
- 无需训练，适配各类视觉语言模型
视觉语言模型中，视觉输入通常被编码为比文本长得多的标记序列，导致视觉标记成为高效推理的主要瓶颈。尽管近期许多方法通过单次评分并剪枝低分标记来缓解该问题，但一次性评分不足，因标记对提示的相关性取决于已保留的证据。受此启发，我们提出DIVE（动态迭代视觉证据构建），一种无需训练的框架，将视觉标记剪枝重构为动态证据构建过程。DIVE反复选择剩余标记中残差条件得分最高的，更新视觉与提示残差以抵消已有解释的证据，并重新评估剩余标记。这一选择-更新-重评循环逐步构建出互补且与提示相关的保留证据集。在八个图像理解基准上的实验表明，DIVE在不同标记预算下均能稳定保持性能；当视觉标记减少88.9%时，平均性能仍保留原模型的98.2%。代码已公开于https://github.com/Zhong-Chenchen/DIVE.git。
原文摘要 · Abstract (English)
Visual inputs in vision-language models (VLMs) are often encoded into substantially longer token sequences than text, making visual tokens a major bottleneck for efficient inference. Abundant recent methods address this bottleneck by scoring token importance and pruning low-scoring tokens in a single pass. However, one-shot scoring is insufficient because a token's prompt-relevant usefulness depends on the evidence already retained. Motivated by this insight, we introduce DIVE (Dynamic Iterative Visual Evidence Construction), a training-free framework that recasts visual-token pruning as dynamic evidence construction. DIVE repeatedly selects the remaining token with the highest residual-conditioned score, updates the visual and prompt residuals to discount the evidence already explained, and re-evaluates the remaining tokens. This select-update-re-evaluate process builds a retained set of complementary, prompt-relevant evidence. Experiments across eight image-understanding benchmarks show that DIVE consistently preserves performance across token budgets. With an 88.9% reduction in visual tokens, DIVE retains 98.2% of the uncompressed model's average performance. Code is available at https://github.com/Zhong-Chenchen/DIVE.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。