LongVU通过自适应压缩实现长视频理解，大幅减少令牌数却保留关键视觉信息。
LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
- 基于跨模态查询与帧间依赖，动态删减冗余时空特征。
- 在时长一小时的视频上表现优异，准确率超越现有方法。
- 适合资源受限场景，轻量模型也能达到顶尖性能。
多模态大语言模型（MLLM）在视频理解方面取得显著进展，但处理长视频仍受上下文长度限制。为此，我们提出LongVU，一种时空自适应压缩机制，在减少视频令牌数量的同时保留关键视觉细节。该方法利用DINOv2特征识别高相似度冗余帧并剔除，结合文本引导的跨模态查询进行选择性帧特征降维，并基于帧间时间依赖关系执行空间令牌压缩。该自适应策略能在有限上下文内高效处理大量帧，几乎不损失视觉信息。LongVU在多种视频理解基准上持续优于现有方法，尤其在时长达一小时的VideoMME和MLVU任务中表现突出。即使使用轻量级LLM，LongVU仍可有效缩放至小模型规模，实现当前最佳视频理解性能。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。