让视频大模型按需聚焦关键片段,提升长视频理解准确性。
Think with Grounding: Curriculum Reinforced Reasoning with Video Grounding for Long Video Understanding
- 设计动态聚焦机制,仅在必要时对相关片段进行定位分析。
- 在三个基准上超越现有最强基线,最高提升达12.3%。
- 适合需要精准理解长视频的科研与工业场景使用。
长视频理解因时空跨度大、多模态线索复杂而困难。现有方法依赖纯文本推理,但在固定视频上下文下易产生幻觉,因重要线索常被冗余信息掩盖。为此,我们提出 Video-TwG 框架,采用新型「思考-定位」范式,使视频大模型能主动决定何时执行按需定位,在交错的文本与视频推理中只聚焦问题相关片段。该框架可端到端训练,无需复杂辅助模块或标注推理轨迹。我们设计两阶段强化课程策略:先在带标注的短视频 GQA 数据集上学习,再扩展至跨领域通用视频 QA 数据以增强泛化能力。针对复杂推理,提出 TwG-GRPO 算法,包含细粒度定位奖励、自验证伪奖励与准确率门控机制。此外,构建了新的 TwG-51K 数据集。在 Video-MME、LongVideoBench 与 MLVU 上实验显示,Video-TwG 持续优于强基线。消融实验证明两阶段课程策略的必要性,并表明 TwG-GRPO 能更好利用未标注数据提升定位质量,减少冗余定位且不牺牲问答性能。
原文摘要 · Abstract (English)
Long video understanding is challenging due to rich and complicated multimodal clues in long temporal range.Current methods adopt reasoning to improve the model's ability to analyze complex video clues in long videos via text-form reasoning.However,the existing literature suffers from the fact that the text-only reasoning under fixed video context may exacerbate hallucinations since detailed crucial clues are often ignored under limited video context length due to the temporal redundancy of long videos.To address this gap,we propose Video-TwG,a curriculum reinforced framework that employs a novel Think-with-Grounding paradigm,enabling video LLMs to actively decide when to perform on-demand grounding during interleaved text-video reasoning, selectively zooming into question-relevant clips only when necessary.Video-TwG can be trained end-to-end in a straightforward manner, without relying on complex auxiliary modules or heavily annotated reasoning tracesIn detail,we design a Two-stage Reinforced Curriculum Strategy, where the model first learns think-with-grounding behavior on a small short-video GQA dataset with grounding labels,and then scales to diverse general QA data with videos of diverse domains to encourage generalization. Further, to handle complex think-with-grounding reasoning for various kinds of data,we propose TwG-GRPO algorithm which features the fine-grained grounding reward, self-confirmed pseudo reward and accuracy-gated mechanism.Finally,we propose to construct a new TwG-51K dataset that facilitates training. Experiments on Video-MME, LongVideoBench, and MLVU show that Video-TwG consistently outperforms strong LVU baselines.Further ablation validates the necessity of our Two-stage Reinforced Curriculum Strategy and shows our TwG-GRPO better leverages diverse unlabeled data to improve grounding quality and reduce redundant groundings without sacrificing QA performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。