通过自监督视频补全提升视频大模型的时序理解能力
MVP: Enhancing Video Large Language Models via Self-supervised Masked Video Prediction
- 设计自监督掩码视频预测任务,强制模型捕捉帧间逻辑关系
- 在多个视频问答数据集上显著提升时序推理准确率
- 适合需要精细视频因果理解的研究者与开发者
基于强化学习的视频大语言模型后训练范式在视觉-语义任务(如字幕生成或视频问答)上取得了显著进展。然而,这类方法主要关注整体内容理解,缺乏对内在时间连贯性和帧间相关性的显式监督,限制了模型对复杂动态和细粒度视觉因果关系的捕捉能力。为此,本文提出一种新的后训练目标——掩码视频预测(MVP):要求模型从一组挑战性干扰项中重建被掩码的连续视频片段,从而迫使模型关注事件的序列逻辑与时间上下文。为支持可扩展训练,我们设计了一个数据合成流水线,可将任意视频语料转化为MVP训练样本,并采用分组相对策略优化(GRPO)与细粒度奖励函数,增强模型对视频上下文与时间特性的理解。全面评估表明,MVP通过直接强化时序推理与因果理解,显著提升了视频推理能力。
原文摘要 · Abstract (English)
Reinforcement learning based post-training paradigms for Video Large Language Models (VideoLLMs) have achieved significant success by optimizing for visual-semantic tasks such as captioning or VideoQA. However, while these approaches effectively enhance perception abilities, they primarily target holistic content understanding, often lacking explicit supervision for intrinsic temporal coherence and inter-frame correlations. This tendency limits the models' ability to capture intricate dynamics and fine-grained visual causality. To explicitly bridge this gap, we propose a novel post-training objective: Masked Video Prediction (MVP). By requiring the model to reconstruct a masked continuous segment from a set of challenging distractors, MVP forces the model to attend to the sequential logic and temporal context of events. To support scalable training, we introduce a scalable data synthesis pipeline capable of transforming arbitrary video corpora into MVP training samples, and further employ Group Relative Policy Optimization (GRPO) with a fine-grained reward function to enhance the model's understanding of video context and temporal properties. Comprehensive evaluations demonstrate that MVP enhances video reasoning capabilities by directly reinforcing temporal reasoning and causal understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。