为视频理解设计高效强化学习框架,提升大模型推理能力。
EasyVideoR1: Easier RL for Video Understanding

- 离线预处理+张量缓存,减少重复解码,吞吐提升1.47倍
- 覆盖11类任务的统一奖励系统,支持模块化扩展
- 适合研究视频理解的模型训练与评估,尤其关注效率与可复现性
基于可验证奖励的强化学习(RLVR)在提升大型语言模型推理能力方面表现卓越。随着模型向原生多模态架构演进,将RLVR拓展至视频理解愈发重要,但受限于视频任务类型多样、高维视觉输入反复解码与预处理带来的计算开销,以及众多敏感超参数下难以复现的评估问题,该方向仍处于空白。现有开源强化学习训练框架虽在文本和图像场景中提供坚实基础设施,却缺乏针对视频模态的系统优化。本文提出 extbf{EasyVideoR1},一个专为大视觉语言模型视频理解任务设计的完整高效强化学习框架。其主要贡献包括:(1)包含离线预处理与张量缓存的全链路视频强化学习训练流程,消除冗余视频解码,实现1.47倍吞吐提升;(2)涵盖11种不同视频与图像任务类型的全面、任务感知奖励系统,支持统一路由与模块化扩展;(3)混合离线-在线数据训练范式,结合高质量轨迹与策略探索,助力更复杂任务的学习;(4)图像-视频联合训练机制,支持独立配置像素预算,使两模态相互增强;(5)异步多基准评估框架,覆盖22个主流视频理解基准,复现准确率与官方报告高度一致。
原文摘要 · Abstract (English)
Reinforcement learning from verifiable rewards (RLVR) has demonstrated remarkable effectiveness in improving the reasoning capabilities of large language models. As models evolve into natively multimodal architectures, extending RLVR to video understanding becomes increasingly important yet remains largely unexplored, due to the diversity of video task types, the computational overhead of repeatedly decoding and preprocessing high-dimensional visual inputs, and the difficulty of reproducible evaluation across numerous sensitive hyperparameters. Existing open-source RL training frameworks provide solid infrastructure for text and image scenarios but lack systematic optimizations tailored for video modality. In this work, we present \textbf{EasyVideoR1}, a complete and efficient reinforcement learning framework specifically designed for training large vision-language models on video understanding tasks. EasyVideoR1 makes the following contributions: (1) a full video RL training pipeline with offline preprocessing and tensor caching that eliminates redundant video decoding and yields a 1.47 $\times$ throughput improvement; (2) a comprehensive, task-aware reward system covering 11 distinct video and image problem types with unified routing and modular extension; (3) a mixed offline-online data training paradigm that combines curated high-quality trajectories with on-policy exploration, benefiting the learning of more challenging tasks; (4) joint image-video training with independently configurable pixel budgets, allowing the two modalities to mutually reinforce each other; and (5) an asynchronous multi-benchmark evaluation framework covering 22 mainstream video understanding benchmarks, with reproduced accuracy closely aligned with officially reported scores.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。