arXiv:2508.07871cs.CV2025-08AAAI被引 20

提出一种自适应剪枝方法,让多模态上下文学习更高效准确。

CATP: Contextually Adaptive Token Pruning for Efficient and Enhanced Multimodal In-Context Learning

  • 通过两阶段渐进式剪枝,动态识别并移除冗余图像令牌
  • 删去77.8%图像令牌后,平均性能提升0.6%,推理延迟降低10.78%
  • 专为多模态上下文学习设计,适合快速领域适配场景

现代大规模视觉语言模型(LVLMs)将每张输入图像转换为远超文本令牌数量的图像令牌。尽管这提升了视觉感知能力,但也带来严重冗余问题。由于图像令牌信息稀疏,大量令牌对推理贡献小却显著增加计算开销。现有图像令牌剪枝方法虽能提升效率且性能损失轻微,但主要集中于单图像任务,忽视了多模态上下文学习(ICL)中更高的冗余度与对效率的更高要求。冗余令牌削弱了多模态ICL在快速领域适配中的优势,并导致性能不稳定。当现有剪枝方法应用于该场景时,会造成显著准确率下降,暴露明显差距与新方法需求。为此,我们提出无训练的上下文自适应令牌剪枝(CATP),该方法通过两阶段渐进剪枝,充分反映输入序列中复杂的跨模态交互。在四个LVLM和八个基准上,移除77.8%图像令牌后,平均性能较原模型提升0.6%,全面超越所有基线;同时推理延迟平均降低10.78%。CATP增强了多模态ICL的实用价值,为交错图像-文本设置下的未来进展奠定基础。

原文摘要 · Abstract (English)

Modern large vision-language models (LVLMs) convert each input image into a large set of tokens that far outnumber the text tokens. Although this improves visual perception, it also introduces severe image token redundancy. Because image tokens contain sparse information, many contribute little to reasoning but greatly increase inference cost. Recent image token pruning methods address this issue by identifying important tokens and removing the rest. These methods improve efficiency with only small performance drops. However, most of them focus on single-image tasks and overlook multimodal in-context learning (ICL), where redundancy is higher and efficiency is more important. Redundant tokens weaken the advantage of multimodal ICL for rapid domain adaptation and lead to unstable performance. When existing pruning methods are applied in this setting, they cause large accuracy drops, which exposes a clear gap and the need for new approaches. To address this, we propose Contextually Adaptive Token Pruning (CATP), a training-free pruning method designed for multimodal ICL. CATP uses two stages of progressive pruning that fully reflect the complex cross-modal interactions in the input sequence. After removing 77.8% of the image tokens, CATP achieves an average performance gain of 0.6% over the vanilla model on four LVLMs and eight benchmarks, clearly outperforming all baselines. At the same time, it improves efficiency by reducing inference latency by an average of 10.78%. CATP strengthens the practical value of multimodal ICL and lays the foundation for future progress in interleaved image-text settings.

多模态学习令牌剪枝高效推理上下文学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。