用强化学习让大模型更准地理解长视频,减少幻觉。
VideoTIR: Accurate Understanding for Long Videos with Efficient Tool-Integrated Reasoning
- 用强化学习引导多级工具高效处理长视频片段
- 在三个数据集上准确率提升显著,推理效率更高
- 适合需要精准长视频理解的研究与应用
现有多模态大模型在长视频理解中常因文本与视觉令牌不平衡导致幻觉。尽管短视觉输入处理良好,但长视频仍需分段处理。当前方法依赖大量精细标注数据且工具调用路径受限。本文提出VideoTIR,利用强化学习驱动多级工具集高效理解长视频。通过零样本与监督微调冷启动策略,引导模型聚焦有意义的视频片段、图像或区域。为减少冗余调用,提出工具动作分组策略优化(TAGPO),通过分步奖励和失败回放复用提升效率。同时构建基于沙盒的轨迹合成框架,生成高质量训练数据。在三个长视频问答基准上实验验证,方法兼具有效性与高效性。
原文摘要 · Abstract (English)
Existing Multimodal Large Language Models (MLLMs) often suffer from hallucinations in long video understanding (LVU), primarily due to the imbalance between textual and visual tokens. Observing that MLLMs handle short visual inputs well, recent LVU works alleviate hallucinations by automatically parsing the vast visual data into manageable segments that can be effectively processed by MLLMs. SFT-based tool-calling methods can serve this purpose, but they typically require vast amounts of fine-grained, high-quality data and suffer from constrained tool-calling trajectories. We propose a novel VideoTIR that leverages Reinforcement Learning (RL) to encourage proper usage of comprehensive multi-level toolkits for efficient long video understanding. VideoTIR explores both Zero-RL and SFT cold-starting to enable MLLMs to retrieve and focus on meaningful video segments/images/regions, enhancing long video understanding both accurately and efficiently. To reduce redundant tool-calling, we propose Toolkit Action Grouped Policy Optimization (TAGPO), which enhances the efficiency of the calling process through stepwise reward assignment and reuse of failed rollouts. Additionally, we develop a sandbox-based trajectory synthesis framework to generate high-quality trajectories data. Extensive experiments on three long-video QA benchmarks demonstrate the effectiveness and efficiency of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。