自适应压缩视频冗余，让大模型看清更长视频。
AdaReTaKe: Adaptive Redundancy Reduction to Perceive Longer for Video-language Understanding
- 根据时间和模型层动态分配压缩比例，灵活减少冗余。
- 将视频处理长度从256帧提升至2048帧，保持关键信息不丢失。
- 无需训练即可提升长视频理解效果，适合大模型优化场景。
多模态大语言模型（MLLMs）虽已革新视频理解能力，但在处理长视频时仍受限于上下文长度。现有方法对视频视觉冗余进行统一压缩，取得一定成效。然而，我们定量分析发现，冗余在时间维度和模型各层间差异显著，需更灵活的压缩策略。为此，提出AdaReTaKe——一种无需训练的方法，通过理论保证在时间与层间动态分配压缩比例，实现视觉冗余的自适应减少。集成至先进MLLMs后，处理能力由256帧扩展至2048帧，同时保留关键信息。在VideoMME、MLVU、LongVideoBench及LVBench数据集上的实验表明，该方法在7B与72B模型上分别超越现有方法2.3%和2.8%，在最长的LVBench上提升达5.9%和6.0%。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have revolutionized video understanding, yet are still limited by context length when processing long videos. Recent methods compress videos by leveraging visual redundancy uniformly, yielding promising results. Nevertheless, our quantitative analysis shows that redundancy varies significantly across time and model layers, necessitating a more flexible compression strategy. We propose AdaReTaKe, a training-free method that flexibly reduces visual redundancy by allocating compression ratios among time and layers with theoretical guarantees. Integrated into state-of-the-art MLLMs, AdaReTaKe improves processing capacity from 256 to 2048 frames while preserving critical information. Experiments on VideoMME, MLVU, LongVideoBench, and LVBench datasets demonstrate that AdaReTaKe outperforms existing methods by 2.3% and 2.8% for 7B and 72B models, respectively, with even greater improvements of 5.9% and 6.0% on the longest LVBench. Our code is available at https://github.com/SCZwangxiao/video-FlexReduc.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。