通过滑窗拼接减少视觉令牌，提升大模型效率与细节理解能力
Window Token Concatenation for Efficient Visual Large Language Models
- 用滑动窗口拼接相邻视觉令牌，降低冗余
- 微调视觉编码器最后几层，使同窗令牌特征更相似
- 适用于需要高效推理和精细视觉理解的场景
为有效减少视觉大语言模型（VLLM）中的视觉令牌数量，本文提出一种新方法——滑窗令牌拼接（Window Token Concatenation, WiCo）。具体而言，采用滑动窗口将空间上相邻的视觉令牌进行拼接。然而，直接拼接可能将不同内容的令牌混合，从而掩盖细粒度信息。为此，我们微调视觉编码器最后几层，使同一窗口内的视觉令牌特征趋于一致。为进一步提升细粒度视觉理解性能，提出WiCo+，在大模型后层对视觉令牌进行分解。该设计既保留了大模型的广域感知优势，又保持少量视觉令牌以实现高效推理。我们在LLaVA-1.5和Shikra基础上，针对粗粒度与细粒度视觉理解任务进行了广泛实验，结果优于现有令牌压缩方法。代码已开源：https://github.com/JackYFL/WiCo。
原文摘要 · Abstract (English)
To effectively reduce the visual tokens in Visual Large Language Models (VLLMs), we propose a novel approach called Window Token Concatenation (WiCo). Specifically, we employ a sliding window to concatenate spatially adjacent visual tokens. However, directly concatenating these tokens may group diverse tokens into one, and thus obscure some fine details. To address this challenge, we propose fine-tuning the last few layers of the vision encoder to adaptively adjust the visual tokens, encouraging that those within the same window exhibit similar features. To further enhance the performance on fine-grained visual understanding tasks, we introduce WiCo+, which decomposes the visual tokens in later layers of the LLM. Such a design enjoys the merits of the large perception field of the LLM for fine-grained visual understanding while keeping a small number of visual tokens for efficient inference. We perform extensive experiments on both coarse- and fine-grained visual understanding tasks based on LLaVA-1.5 and Shikra, showing better performance compared with existing token reduction projectors. The code is available: https://github.com/JackYFL/WiCo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。