视频大模型推理加速新框架，仅用25%视觉标记实现99.6%性能
Video Compression Commander: Plug-and-Play Inference Acceleration for Video Large Language Models
- 按帧独特性动态调整压缩强度，避免信息丢失
- 仅用25%视觉标记，保留99.6%原模型性能，推理延迟降低70.8%
- 可插拔设计，兼容主流视频大模型与压缩方法
视频大语言模型（VideoLLM）在视频理解任务中表现优异，但因视觉标记数量庞大导致计算复杂度呈二次增长。我们对现有视频标记压缩方法的系统分析揭示两个关键问题：(i) 忽视帧间视觉信号差异，造成信息损失；(ii) 受限于实现方式，难以适配现代架构或高效算子。为此，我们提炼出三项设计原则，并提出可即插即用的推理加速框架「Video Compression Commander」（VidCom2）。通过量化每帧的独特性，VidCom2自适应调节压缩强度，在有效减少冗余的同时保留关键信息。在多个视频大模型和基准测试上，实验结果表明其兼具卓越性能与效率：仅使用25%视觉标记，便在LLaVA-OV上达到99.6%的原始性能，且推理延迟降低70.8%。值得注意的是，其帧压缩调节策略可与其它压缩方法结合，进一步提升效果。代码已开源。
原文摘要 · Abstract (English)
Video large language models (VideoLLM) excel at video understanding, but face efficiency challenges due to the quadratic complexity of abundant visual tokens. Our systematic analysis of token compression methods for VideoLLMs reveals two critical issues: (i) overlooking distinctive visual signals across frames, leading to information loss; (ii) suffering from implementation constraints, causing incompatibility with modern architectures or efficient operators. To address these challenges, we distill three design principles for VideoLLM token compression and propose a plug-and-play inference acceleration framework "Video Compression Commander" (VidCom2). By quantifying each frame's uniqueness, VidCom2 adaptively adjusts compression intensity across frames, effectively preserving essential information while reducing redundancy in video sequences. Extensive experiments across various VideoLLMs and benchmarks demonstrate the superior performance and efficiency of our VidCom2. With only 25% visual tokens, VidCom2 achieves 99.6% of the original performance on LLaVA-OV while reducing 70.8% of the LLM generation latency. Notably, our Frame Compression Adjustment strategy is compatible with other token compression methods to further improve their performance. Our code is available at https://github.com/xuyang-liu16/VidCom2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。