用少量注册符压缩视觉令牌，大幅提速模型训练与推理。
Efficient Vision-Language Models by Summarizing Visual Tokens into Compact Registers
- 用可学习的注册符在语言塔前几层汇总视觉信息
- 8个注册符仅占原视觉令牌1%，准确率下降不足4%
- 适合需要高效推理的部署场景，实现3.3倍加速
近期视觉语言模型（VLMs）的发展拓展了其在真实场景中的应用潜力，使模型能够对图像进行复杂推理。在广泛使用的全自回归Transformer架构模型（如LLaVA）中，投影后的视觉令牌被前置到文本令牌前。通常视觉令牌数量远超提示令牌，导致训练和推理时计算开销显著增加。本文提出视觉紧凑令牌注册机制（Victor），通过将视觉令牌压缩为更少的注册令牌来减少视觉令牌数量。Victor在视觉令牌后添加若干可学习的注册令牌，并利用语言塔的前几层将视觉信息归纳至这些注册符中；在经过这几层后，所有原始视觉令牌被丢弃，从而显著提升训练与推理效率。该方法易于实现，新增可训练参数极少，对模型性能影响微小。实验表明，仅使用8个视觉注册符（约原令牌的1%），准确率下降低于4%，总训练时间减少43%，推理吞吐量提升3.3倍。
原文摘要 · Abstract (English)
Recent advancements in vision-language models (VLMs) have expanded their potential for real-world applications, enabling these models to perform complex reasoning on images. In the widely used fully autoregressive transformer-based models like LLaVA, projected visual tokens are prepended to textual tokens. Oftentimes, visual tokens are significantly more than prompt tokens, resulting in increased computational overhead during both training and inference. In this paper, we propose Visual Compact Token Registers (Victor), a method that reduces the number of visual tokens by summarizing them into a smaller set of register tokens. Victor adds a few learnable register tokens after the visual tokens and summarizes the visual information into these registers using the first few layers in the language tower of VLMs. After these few layers, all visual tokens are discarded, significantly improving computational efficiency for both training and inference. Notably, our method is easy to implement and requires a small number of new trainable parameters with minimal impact on model performance. In our experiment, with merely 8 visual registers--about 1% of the original tokens--Victor shows less than a 4% accuracy drop while reducing the total training time by 43% and boosting the inference throughput by 3.3X.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。