提出分层视频压缩技术，让大模型高效处理长视频。
VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
- 用分层压缩减少视频上下文计算量，压缩比达1/50
- 在1万帧视频中实现99.1%准确率，性能领先开源模型
- 适合需要处理长视频的多模态大模型研究者
长视频建模对多模态大语言模型至关重要，使其能够处理电影、在线视频流等。尽管取得进展，但高效理解极长视频上下文仍具挑战。本文从模型架构、训练数据、训练策略和评估基准四个方面解决该问题。首先，提出一种新型分层视频标记压缩（HiCo）方法，利用长视频中的视觉冗余，将视频上下文从片段级压缩至视频级，显著降低计算量，压缩比约1/50，几乎无性能损失。其次，引入多阶段由短到长的学习方案、大规模真实长视频数据集LongVid，以及具有挑战性的“多跳针在视频堆”（Multi-Hop Needle-In-A-Video-Haystack, NIAH）基准。最后，构建强大的视频多模态大模型VideoChat-Flash，在2B和7B规模下均在主流长短视频基准上表现领先。其在开放源代码模型中于NIAH任务上达到99.1%准确率（10,000帧）。
原文摘要 · Abstract (English)
Long-context video modeling is critical for multimodal large language models (MLLMs), enabling them to process movies, online video streams, and so on. Despite its advances, handling long videos remains challenging due to the difficulty in efficiently understanding the extremely long video context. This paper aims to address this issue from aspects of model architecture, training data, training strategy and evaluation benchmark. First, we propose a novel Hierarchical video token Compression (HiCo) method, which leverages visual redundancy in long videos to compress long video context from Clip-level to Video-level, reducing the computation significantly while preserving essential details, achieving an extreme compression ratio of approximately 1/50 with almost no performance loss. Second, we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of real-world long videos named LongVid, and a challenging ``Multi-Hop Needle-In-A-Video-Haystack'' benchmark. Finally, we build a powerful video MLLM named VideoChat-Flash, which shows a leading performance on both mainstream long and short video benchmarks at the 2B and 7B model scale. It first gets 99.1% accuracy over 10,000 frames in NIAH among open-source models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。