用强化学习提升视频大模型的推理能力,更省数据、效果更好
Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency
- 设计双奖励机制,分别指导语义和时序推理
- 仅用少量数据就超越监督微调和现有强化学习方法
- 适合研究视频理解与大模型推理的学者和工程师
理解具有复杂语义和长时序依赖的真实世界视频仍是计算机视觉中的基础挑战。近年来,多模态大语言模型(MLLM)在视觉-语言任务中展现出强大能力,而强化学习微调(RLT)进一步提升了其推理性能。本文探索将RLT作为后训练策略,增强MLLM在视频理解方面的推理能力。基于组相对策略优化(GRPO)框架,提出一种双奖励形式,通过离散与连续奖励信号同时监督语义和时序推理。为实现有效的偏好优化,引入基于重复推理的方差感知数据选择策略,识别提供有效学习信号的样本。在八个代表性视频理解任务上评估,包括VideoQA、时序视频定位和基于视觉定位的VideoQA。所提方法始终优于监督微调及现有RLT基线,在显著减少训练数据的前提下取得更优表现。结果凸显了奖励设计与数据选择在推动以推理为中心的视频理解中的关键作用。初始代码发布两个月后已更新,包含优化后的奖励机制和新增数据集,最新版本见https://github.com/appletea233/Temporal-R1。
原文摘要 · Abstract (English)
Understanding real-world videos with complex semantics and long temporal dependencies remains a fundamental challenge in computer vision. Recent progress in multimodal large language models (MLLMs) has demonstrated strong capabilities in vision-language tasks, while reinforcement learning tuning (RLT) has further improved their reasoning abilities. In this work, we explore RLT as a post-training strategy to enhance the video-specific reasoning capabilities of MLLMs. Built upon the Group Relative Policy Optimization (GRPO) framework, we propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals. To facilitate effective preference-based optimization, we introduce a variance-aware data selection strategy based on repeated inference to identify samples that provide informative learning signals. We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA. Our method consistently outperforms supervised fine-tuning and existing RLT baselines, achieving superior performance with significantly less training data. These results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs. Notably, The initial code release (two months ago) has now been expanded with updates, including optimized reward mechanisms and additional datasets. The latest version is available at https://github.com/appletea233/Temporal-R1 .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。