动态压缩视觉令牌，让大模型更高效处理多图视频。
LLaVA-Zip: Adaptive Visual Token Compression with Intrinsic Image Information
- 基于LLaVA-1.5设计动态特征图压缩机制，按需减少视觉令牌数量。
- 在不同图像数量下性能提升显著，最高提升达12.7%（对比基线）。
- 适合资源受限的学术研究，也适用于工业场景的数据增强。
采用指令遵循数据的多模态大语言模型（MLLMs），如LLaVA，在产业界已取得显著进展。其主要局限在于视觉令牌占用了大语言模型（LLM）最大令牌限制的很大比例，导致在包含多张图像或视频的提示中计算开销增加、性能下降。产业界常通过提升算力来缓解此问题，但在算力有限的学术环境中不具可行性。本研究提出基于LLaVA-1.5的动态特征图压缩（DFMR）方法，以应对视觉令牌过载挑战。DFMR动态压缩视觉令牌，释放令牌容量。实验结果表明，将DFMR集成到LLaVA-1.5中，在不同视觉令牌长度下均显著提升性能，为在资源受限的学术环境中扩展LLaVA处理多图像和视频场景提供了可行方案，也可应用于工业领域进行数据增强，缓解持续预训练阶段开放域图文配对数据集稀缺的问题。
原文摘要 · Abstract (English)
Multi-modal large language models (MLLMs) utilizing instruction-following data, such as LLaVA, have achieved great progress in the industry. A major limitation in these models is that visual tokens consume a substantial portion of the maximum token limit in large language models (LLMs), leading to increased computational demands and decreased performance when prompts include multiple images or videos. Industry solutions often mitigate this issue by increasing computational power, but this approach is less feasible in academic environments with limited resources. In this study, we propose Dynamic Feature Map Reduction (DFMR) based on LLaVA-1.5 to address the challenge of visual token overload. DFMR dynamically compresses the visual tokens, freeing up token capacity. Our experimental results demonstrate that integrating DFMR into LLaVA-1.5 significantly improves the performance of LLaVA in varied visual token lengths, offering a promising solution for extending LLaVA to handle multi-image and video scenarios in resource-constrained academic environments and it can also be applied in industry settings for data augmentation to help mitigate the scarcity of open-domain image-text pair datasets in the continued pretraining stage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。