arXiv:2602.18846cs.CVcs.AI2026-02

提出双阶段压缩框架,让视觉语言模型更高效且不丢精度。

DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

  • 先压缩视觉编码器输出,再在语言模型中按语义重要性逐步丢弃视觉标记。
  • 在LLaVA-1.5上实现89%标记减少仍保持97.6%准确率,视频模型超基准100%准确。
  • 适用于追求高效率与高精度平衡的多模态应用,如视频理解与推理系统。

视觉语言模型(VLMs)虽具备强大跨模态理解与推理能力,但因密集视觉标记化导致计算成本高昂。现有高效方法或合并冗余视觉标记,或在语言骨干中逐步丢弃,常以牺牲精度换取速度。本文提出DUET-VLM,一种即插即用的双阶段压缩框架:首先对视觉编码器输出进行仅保留信息的冗余感知压缩;随后在语言骨干中逐层、基于文本引导地剔除不重要视觉标记,实现协同压缩。该机制可在大幅降低标记数的同时保持关键语义。在LLaVA-1.5-7B上,本方法在减少67%标记时仍保持超过99%的基线准确率,即使在89%压缩下仍达97.6%以上。训练阶段采用此压缩策略后,准确率分别达99.7%(67%压缩)和97.6%(89%压缩),优于现有最先进视觉标记压缩方法。集成至Video-LLaVA-7B时,甚至在53.1%标记减少下超越基线(>100%准确率),极端93.4%压缩下仍保持97.6%准确率。结果表明,通过端到端训练使用DUET-VLM,可在不损失准确性的前提下,适应极简视觉输入,生成紧凑而语义丰富的表示,且维持相同计算开销。

原文摘要 · Abstract (English)

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.

视觉语言模型标记压缩高效推理多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。