不训练即可加速多模态大模型，通过视觉关键信息保留与文本引导合并实现
VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration
- 基于全局局部视角选择关键视觉标记，保留核心信息
- 利用文本提示引导上下文感知的标记合并，减少冗余
- 无需重新训练，适配多种图像视频任务，部署更高效
多模态大语言模型因视觉标记过多导致计算开销高，尤其在高分辨率和视频场景中更为严重。现有标记压缩方法通常仅针对单一模块，且忽视文本对齐，常导致性能下降。本文提出VisionTrim，一种无需训练的统一加速框架，包含两个即插即用模块：1）主导视觉标记选择（DVTS），通过全局-局部视图保留关键视觉标记；2）文本引导视觉补全（TGVC），借助文本线索实现上下文感知的标记合并。在多种图像与视频多模态基准上的实验表明，VisionTrim显著优于现有方法，推动了实际应用中的多模态大模型部署。代码已开源：https://github.com/hanxunyu/VisionTrim。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) suffer from high computational costs due to excessive visual tokens, particularly in high-resolution and video-based scenarios. Existing token reduction methods typically focus on isolated pipeline components and often neglect textual alignment, leading to performance degradation. In this paper, we propose VisionTrim, a unified framework for training-free MLLM acceleration, integrating two effective plug-and-play modules: 1) the Dominant Vision Token Selection (DVTS) module, which preserves essential visual tokens via a global-local view, and 2) the Text-Guided Vision Complement (TGVC) module, which facilitates context-aware token merging guided by textual cues. Extensive experiments across diverse image and video multimodal benchmarks demonstrate the performance superiority of our VisionTrim, advancing practical MLLM deployment in real-world applications. The code is available at: https://github.com/hanxunyu/VisionTrim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。