通过自顶向下压缩，大幅减少视觉令牌数量并保持性能。
Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning
- 提出自顶向下压缩机制，高效保留关键视觉信息。
- 压缩75%~95%视觉令牌，12个基准上表现相当或更优。
- 适合追求高效视觉指令理解的模型部署场景。
视觉指令微调旨在使大语言模型理解视觉世界，核心挑战在于建立有效的视觉到语言映射。现有方法常面临精度与效率的难以兼顾。本文提出LLaVA-Meteor，引入新型自顶向下压缩范式，在不损失核心信息的前提下策略性压缩视觉令牌。具体而言，设计可训练的快速全局融合模块，基于高效选择性状态空间算子对齐特征空间，使每个令牌以低成本感知全局视觉上下文与指令偏好。同时采用局部到单个扫描方式有效捕捉局部依赖，增强视觉建模能力。为降低计算开销，提出视觉原生选择机制，由视觉与原生专家独立评估令牌重要性，再聚合保留最关键的子集。大量实验表明，该方法在12个基准上将视觉令牌减少75%~95%，仍保持相当或更优性能，显著提升效率。
原文摘要 · Abstract (English)
Visual instruction tuning aims to enable large language models to comprehend the visual world, with a pivotal challenge lying in establishing an effective vision-to-language projection. However, existing methods often grapple with the intractable trade-off between accuracy and efficiency. In this paper, we present LLaVA-Meteor, a novel approach designed to break this deadlock, equipped with a novel Top-Down Compression paradigm that strategically compresses visual tokens without compromising core information. Specifically, we construct a trainable Flash Global Fusion module based on efficient selective state space operators, which aligns the feature space while enabling each token to perceive holistic visual context and instruction preference at low cost. Furthermore, a local-to-single scanning manner is employed to effectively capture local dependencies, thereby enhancing the model's capability in vision modeling. To alleviate computational overhead, we explore a Visual-Native Selection mechanism that independently assesses token significance by both the visual and native experts, followed by aggregation to retain the most critical subset. Extensive experiments show that our approach reduces visual tokens by 75--95% while achieving comparable or superior performance across 12 benchmarks, significantly improving efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。