提升视频多模态大模型对长视频与细节的感知能力
InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
- 用密集视觉标注+直接偏好优化增强模型理解力
- 通过自适应分层压缩实现紧凑时空表征，支持6倍长视频输入
- 适合研究视频理解、长序列建模的开发者与研究人员
本文旨在通过长而丰富的上下文（LRC）建模提升视频多模态大语言模型（MLLM）性能。为此，我们推出了InternVideo2.5，重点增强原模型对细粒度细节的感知能力及对长视频时序结构的捕捉能力。具体方法包括：利用密集视觉任务标注通过直接偏好优化注入模型，并采用自适应分层令牌压缩生成紧凑的时空表示。实验表明，该设计显著提升了主流视频理解基准（短/长视频）上的表现，使模型可处理至少6倍于原始长度的视频输入，并掌握目标追踪、分割等专项视觉能力。本工作强调了多模态上下文丰富性（长度与精细度）对激发MLLM内在能力（专注力与记忆力）的重要性，为未来视频MLLM研究提供新思路。代码与模型开源于https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5。
原文摘要 · Abstract (English)
This paper aims to improve the performance of video multimodal large language models (MLLM) via long and rich context (LRC) modeling. As a result, we develop a new version of InternVideo2.5 with a focus on enhancing the original MLLMs' ability to perceive fine-grained details and capture long-form temporal structure in videos. Specifically, our approach incorporates dense vision task annotations into MLLMs using direct preference optimization and develops compact spatiotemporal representations through adaptive hierarchical token compression. Experimental results demonstrate this unique design of LRC greatly improves the results of video MLLM in mainstream video understanding benchmarks (short & long), enabling the MLLM to memorize significantly longer video inputs (at least 6x longer than the original), and master specialized vision capabilities like object tracking and segmentation. Our work highlights the importance of multimodal context richness (length and fineness) in empowering MLLM's innate abilites (focus and memory), providing new insights for future research on video MLLM. Code and models are available at https://github.com/OpenGVLab/InternVideo/tree/main/InternVideo2.5
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。