用新方法提升视频大模型的推理能力，效果显著。
DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
- 将GRPO转为回归任务，不用裁剪等保护机制
- 在多个视频推理基准上性能明显提升
- 适合研究视频理解与强化学习结合的学者
近期研究证明了基于强化学习的后训练能有效提升大语言模型的推理能力。特别是基于组归一化奖励的群组相对策略优化（GRPO）在采用类似PPO的强化学习算法方面表现优异。然而，GRPO在视频大语言模型（VideoLLMs）中的有效性尚未充分探索。本文研究了GRPO并发现两个阻碍有效学习的问题：(1) 对防护机制的依赖，(2) 优势信号消失。为此，我们提出DeepVideo-R1，使用改进的回归式GRPO（Reg-GRPO）和难度感知数据增强进行训练。Reg-GRPO将GRPO损失函数重构为直接预测优势的回归任务，无需裁剪、最小值等操作，使模型更直接地对齐优势信号，从而偏好更优输出。难度感知数据增强策略通过调整输入提示/视频以达到可解难度水平，实现多样化的奖励信号。实验结果表明，该方法在多个基准测试中显著提升了视频推理性能。
原文摘要 · Abstract (English)
Recent works have demonstrated the effectiveness of reinforcement learning (RL)-based post-training for enhancing the reasoning capabilities of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) has shown impressive success using a PPO-style reinforcement learning algorithm with group-normalized rewards. However, the effectiveness of GRPO in Video Large Language Models (VideoLLMs) remains underexplored. In this paper, we explore GRPO and identify two issues that hinder effective learning: (1) reliance on safeguards, and (2) vanishing advantage. To mitigate these challenges, we propose DeepVideo-R1, a video large language model trained with Reg-GRPO (Regressive GRPO) and difficulty-aware data augmentation. Reg-GRPO reformulates the GRPO loss function as a regression task that directly predicts the advantage in GRPO, eliminating the need for safeguards such as clipping and min operations. This directly aligns the model with the advantages, providing guidance to prefer better outputs. The difficulty-aware data augmentation strategy augments input prompts/videos to target solvable difficulty levels, enabling diverse reward signals. Our experimental results show that our approach significantly improves video reasoning performance across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。