用强化学习提升视频理解能力,效果超GPT-4o。
Video-R1: Reinforcing Video Reasoning in MLLMs
- 设计T-GRPO算法,让模型更好利用视频时间信息
- 混合图像与视频数据训练,提升空间推理准确率至37.1%
- 首个专注视频推理的R1范式探索,适合多模态研究者
受DeepSeek-R1通过规则强化学习激发推理能力的启发,我们提出Video-R1,首次系统探索在多模态大语言模型(MLLMs)中应用R1范式以激励视频推理。直接使用GRPO算法进行视频推理训练面临两大挑战:(i) 缺乏对视频时序信息的建模能力,(ii) 高质量视频推理数据稀缺。为此,我们提出T-GRPO算法,鼓励模型利用视频中的时间信息进行推理;同时,不依赖单一视频数据,引入高质量图像推理数据辅助训练。我们构建了两个数据集:Video-R1-CoT-165k用于SFT冷启动,Video-R1-260k用于强化学习训练,均包含图像与视频数据。实验表明,Video-R1在VideoMMMU、VSI-Bench等视频推理基准上表现显著提升,也在MVBench和TempCompass等通用视频基准上取得优异成绩。特别地,Video-R1-7B在视频空间推理基准VSI-Bench上达到37.1%准确率,超越商用模型GPT-4o。所有代码、模型与数据已开源。
原文摘要 · Abstract (English)
Inspired by DeepSeek-R1's success in eliciting reasoning abilities through rule-based reinforcement learning (RL), we introduce Video-R1 as the first attempt to systematically explore the R1 paradigm for incentivizing video reasoning within multimodal large language models (MLLMs). However, directly applying RL training with the GRPO algorithm to video reasoning presents two primary challenges: (i) a lack of temporal modeling for video reasoning, and (ii) the scarcity of high-quality video-reasoning data. To address these issues, we first propose the T-GRPO algorithm, which encourages models to utilize temporal information in videos for reasoning. Additionally, instead of relying solely on video data, we incorporate high-quality image-reasoning data into the training process. We have constructed two datasets: Video-R1-CoT-165k for SFT cold start and Video-R1-260k for RL training, both comprising image and video data. Experimental results demonstrate that Video-R1 achieves significant improvements on video reasoning benchmarks such as VideoMMMU and VSI-Bench, as well as on general video benchmarks including MVBench and TempCompass, etc. Notably, Video-R1-7B attains a 37.1% accuracy on video spatial reasoning benchmark VSI-bench, surpassing the commercial proprietary model GPT-4o. All code, models, and data are released in: https://github.com/tulerfeng/Video-R1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。