通过优化视觉信号利用，让视频大模型在只保留10%数据时仍保持接近原性能。
Tango: Taming Visual Signals for Efficient Video Large Language Models
- 引入多样性策略改进注意力选token，更好捕捉多模态分布
- 用时空旋转位置编码保留几何结构，避免聚类碎片化
- 适合追求高效视频理解的开发者，尤其在资源受限场景
令牌剪枝已成为构建高效视频大语言模型（Video LLMs）的主流方法。本文重新审视并推进了两种主要的令牌剪枝范式：基于注意力的选择与基于相似性的聚类。研究发现现有方法存在两个关键缺陷：(1) 传统的 top-k 选择策略未能充分考虑注意力分布，而注意力分布常具有空间多模态和长尾特性；(2) 直接基于相似性的聚类常导致碎片化簇，池化后产生失真表征。为此，我们提出 Tango 框架，旨在优化视觉信号的利用。Tango 集成多样性驱动策略以提升基于注意力的令牌选择，并引入时空旋转位置编码（ST-RoPE），通过局部性先验保留几何结构。在多种 Video LLM 和视频理解基准上的综合实验表明该方法有效且具备强泛化能力。值得注意的是，当仅保留 10% 的视频令牌时，Tango 在 LLaVA-OV 上仍保持 98.9% 的原始性能，同时实现 1.88× 的推理加速。
原文摘要 · Abstract (English)
Token pruning has emerged as a mainstream approach for developing efficient Video Large Language Models (Video LLMs). This work revisits and advances the two predominant token-pruning paradigms: attention-based selection and similarity-based clustering. Our study reveals two critical limitations in existing methods: (1) conventional top-k selection strategies fail to fully account for the attention distribution, which is often spatially multi-modal and long-tailed in magnitude; and (2) direct similarity-based clustering frequently generates fragmented clusters, resulting in distorted representations after pooling. To address these bottlenecks, we propose Tango, a novel framework designed to optimize the utilization of visual signals. Tango integrates a diversity-driven strategy to enhance attention-based token selection, and introduces Spatio-temporal Rotary Position Embedding (ST-RoPE) to preserve geometric structure via locality priors. Comprehensive experiments across various Video LLMs and video understanding benchmarks demonstrate the effectiveness and generalizability of our approach. Notably, when retaining only 10% of the video tokens, Tango preserves 98.9% of the original performance on LLaVA-OV while delivering a 1.88$\times$ inference speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。