用语义连通分量压缩视频令牌，提升长视频理解效率
LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMs
- 基于语义连通分量将令牌分组，覆盖完整语义区域
- 在低保留率下仍保持优异性能，长视频问答准确率领先
- 无需训练，适合部署于各类视频大模型
本文提出 LLaVA-Scissor，一种面向视频多模态大语言模型的无训练令牌压缩策略。现有方法多依赖注意力得分压缩令牌，难以覆盖全部语义区域且易产生冗余。我们提出利用语义连通分量（SCC）将令牌分配至不同语义区域，确保语义完整性。该方法构建了时空双域的两步压缩策略，在空间与时间维度均使用 SCC，以非重叠语义令牌表示整段视频。我们在多个视频理解基准上评估其压缩能力，包括视频问答、长视频理解及综合多选题任务。实验表明，LLaVA-Scissor 在多种视频理解任务中优于现有压缩方法，尤其在低令牌保留率下表现更优。
原文摘要 · Abstract (English)
In this paper, we present LLaVA-Scissor, a training-free token compression strategy designed for video multimodal large language models. Previous methods mostly attempt to compress tokens based on attention scores, but fail to effectively capture all semantic regions and often lead to token redundancy. Differently, we propose to leverage the Semantic Connected Components (SCC) approach that assigns tokens to distinct semantic regions within the token set, ensuring comprehensive semantic coverage. The outcome is a two-step spatio-temporal token compression strategy that utilizes SCC in both spatial and temporal domains. This strategy can effectively compress tokens by representing the entire video with a set of non-overlapping semantic tokens. We conduct extensive evaluations of the token compression capabilities of LLaVA-Scissor across diverse video understanding benchmarks, including video question answering, long video understanding, and comprehensive multi-choices benchmarks. Experimental results show that the proposed LLaVA-Scissor outperforms other token compression methods, achieving superior performance in various video understanding benchmarks, particularly at low token retention ratios. Project page: https://github.com/HumanMLLM/LLaVA-Scissor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。