自适应剪裁视觉令牌，高效处理长上下文多图任务
TrimTokenator-LC: Towards Adaptive Visual Token Pruning for Large Multimodal Models with Long Contexts
- 按图像内多样性和图像间差异性动态分配剪裁预算
- 可减少80%视觉令牌，保持长上下文任务性能
- 适合需要高效处理多图长序列的大型多模态模型
大型多模态模型（LMMs）在各类任务中表现优异，通常将视觉输入编码为原始模型的令牌序列，再与文本令牌拼接后由语言模型联合处理。然而，视觉令牌数量增长显著增加了推理开销。视觉令牌剪裁成为有前景的解决方案。现有方法常忽视包含多图像的长上下文场景。本文分析了此类场景下视觉令牌剪裁的挑战，提出一种自适应剪裁方法。我们将冗余分解为图像内和图像间两部分，通过图像内多样性与图像间差异性量化，并据此动态分配剪裁预算。方法分两阶段：第一阶段为每张图像分配内容感知的令牌预算，贪心选择最具代表性令牌；第二阶段进行全局多样性过滤形成候选池，并采用帕累托选择平衡多样性与文本对齐。大量实验表明，该方法可在长上下文设置中减少高达80%的视觉令牌，同时保持性能。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) have proven effective on various tasks. They typically encode visual inputs into Original Model sequences of tokens, which are then concatenated with textual tokens and jointly processed by the language model. However, the growing number of visual tokens greatly increases inference cost. Visual token pruning has emerged as a promising solution. However, existing methods often overlook scenarios involving long context inputs with multiple images. In this paper, we analyze the challenges of visual token pruning in long context, multi-image settings and introduce an adaptive pruning method tailored for such scenarios. We decompose redundancy into intra-image and inter-image components and quantify them through intra-image diversity and inter-image variation, which jointly guide dynamic budget allocation. Our approach consists of two stages. The intra-image stage allocates each image a content-aware token budget and greedily selects its most representative tokens. The inter-image stage performs global diversity filtering to form a candidate pool and then applies a Pareto selection procedure that balances diversity with text alignment. Extensive experiments show that our approach can reduce up to 80% of visual tokens while maintaining performance in long context settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。