让AI像人一样看视频推理,通过工具调用实现精准长视频理解。
Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
- 用视觉工具动态采样帧,结合多模态思维链提升跨模态交互
- 在11个基准上超越现有方法,长视频问答与定位任务表现最优
- 专为复杂视频设计,适合需要深度时序理解的研究与应用
多模态大语言模型的视频推理能力对视频问答、时间定位等下游任务至关重要。尽管已有研究探索文本思维链(CoT)推理,但这些方法常因跨模态交互有限且易产生幻觉,尤其在长视频或长推理链中表现不佳。为此,我们提出端到端的代理式视频推理框架VITAL,通过视觉工具按需密集采样新帧,并生成多模态思维链以实现精准长视频推理。我们发现时间定位与问答任务相互促进,因此构建了两个高质量多任务数据集:用于监督微调的MTVR-CoT-72k和用于强化学习的MTVR-RL-110k。此外,提出难度感知组相对策略优化算法(DGRPO),缓解多任务强化学习中的难度不平衡问题。在11个挑战性视频理解基准上的大量实验表明,VITAL展现出先进推理能力,在视频问答与时间定位任务中显著优于现有方法,尤其在长视频场景下优势明显。代码已公开于https://zhang9302002.github.io/thinkingwithvideos-page/。
原文摘要 · Abstract (English)
The video reasoning ability of multimodal large language models (MLLMs) is crucial for downstream tasks like video question answering and temporal grounding. While recent approaches have explored text-based chain-of-thought (CoT) reasoning for MLLMs, these methods often suffer from limited cross-modal interaction and increased hallucination, especially with longer videos or reasoning chains. To address these challenges, we propose Video Intelligence via Tool-Augmented Learning (VITAL), a novel end-to-end agentic video reasoning framework. With a visual toolbox, the model can densely sample new video frames on demand and generate multimodal CoT for precise long video reasoning. We observe that temporal grounding and question answering are mutually beneficial for video understanding tasks. Therefore, we construct two high-quality multi-task video reasoning datasets MTVR-CoT-72k for supervised fine-tuning and MTVR-RL-110k for reinforcement learning. Moreover, we propose a Difficulty-aware Group Relative Policy Optimization algorithm (DGRPO) to mitigate difficulty imbalance in multi-task reinforcement learning. Extensive experiments on 11 challenging video understanding benchmarks demonstrate the advanced reasoning ability of VITAL, outperforming existing methods in video question answering and temporal grounding tasks, especially in long video scenarios. Code is available at https://zhang9302002.github.io/thinkingwithvideos-page/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。