用可学习压缩技术让模型高效理解长达数小时的视频。
Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understanding
- 通过自监督学习生成紧凑且完整的视频标记，提升长视频理解效率。
- 仅30亿参数即超越多数70亿参数模型，在多基准上表现更优。
- 适合需要处理超长视频的场景，如视频检索、内容分析等应用。
尽管已有先进的标记压缩技术，现有多模态大语言模型仍难以处理长达一小时的视频理解任务。本文提出Video-XL-Pro，一种基于可重构标记压缩（ReCoT）的高效长视频理解方法。ReCoT引入两个关键组件：(i) 动态标记合成器（DTS），通过学习标记内关系从静态图像标记生成伪视频标记，并用于掩码视频建模；(ii) 语义引导掩码（SGM），自适应地屏蔽冗余视觉标记以促进更有效的重建学习。为提升多模态大模型微调训练效率，我们设计了视频专用数据集剪枝策略，并提出简单但查询感知的选择器，使模型能精准定位与查询相关的视频标记。仅需30亿参数，Video-XL-Pro在多个长视频理解基准上优于多数在更大数据集上训练的70亿参数模型。此外，其可在单块A100 GPU上处理超过8,000帧视频，同时保持高质量性能。
原文摘要 · Abstract (English)
Despite advanced token compression techniques, existing multimodal large language models (MLLMs) still struggle with hour-long video understanding. In this work, we propose Video-XL-Pro, an efficient method for extremely long video understanding, built upon Reconstructive Compression of Tokens (ReCoT), a learnable module that leverages self-supervised learning to generate comprehensive and compact video tokens. ReCoT introduces two key components: (i) Dynamic Token Synthesizer (DTS): DTS generates pseudo-video tokens from static image tokens by learning intra-token relationships, which are then used in masked video modeling. (ii) Semantic-Guided Masking (SGM): SGM adaptively masks redundant visual tokens to facilitate more effective reconstructive learning. To improve training efficiency in MLLMs fine-tuning, we introduce a video-specific dataset pruning strategy and design a simple yet Query-aware Selector that enables the model to precisely locate query-relevant video tokens. With only 3B parameters, Video-XL-Pro outperforms most 7B models trained on larger datasets across multiple long video understanding benchmarks. Moreover, it can process over 8K frames on a single A100 GPU while maintaining high-quality performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。