通过粗到精策略压缩视觉令牌，兼顾效率与性能提升。
FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression
- 采用视觉引导采样去除低信息密度区域冗余
- 引入文本引导采样保留与指令强相关的视觉信息
- 适合需要高分辨率图像输入的多模态任务应用
多模态大模型的发展表明，高分辨率图像输入对模型能力至关重要，尤其在细粒度任务中。然而，高分辨率图像会导致输入到语言模型的视觉令牌数量呈二次增长，带来显著计算开销。现有方法虽尝试通过视觉令牌压缩提升效率，但常以性能为代价。本文认为，消除视觉冗余可同时提升效率与性能。提出一种粗到精的视觉令牌压缩方法：利用视觉引导采样器压缩低信息密度区域，用文本引导采样器选择与用户指令强相关的视觉令牌。该方法在多个评估数据集上验证了有效性，实现了效率与性能的双重提升。
原文摘要 · Abstract (English)
Recent advances on Multi-modal Large Language Models have demonstrated that high-resolution image input is crucial for model capabilities, especially for fine-grained tasks. However, high-resolution images lead to a quadratic increase in the number of visual tokens input into LLMs, resulting in significant computational costs. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We argue that removing visual redundancy can simultaneously improve both efficiency and performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user instructions.With these two modules, the proposed FocusLLaVA achieves improvements in both efficiency and performance. We validate the effectiveness of our approach on a wide range of evaluation datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。