让大模型更准定位视频时间点，提升细节理解能力。
MUSEG: Reinforcing Video Temporal Understanding via Timestamp-Aware Multi-Segment Grounding
- 用带时间戳的多片段对齐，引导模型精准匹配视频段落
- 在时间定位和时序问答任务上显著超越现有方法
- 适合需要精细时间推理的视频理解场景
视频时间理解对多模态大语言模型（MLLMs）推理视频事件至关重要。尽管通用视频理解取得进展，当前MLLMs在细粒度时间推理上仍表现不佳。虽有研究尝试用强化学习（RL）改进，但现有方法在时间敏感任务上性能有限。本文提出MUSEG，一种基于强化学习的新方法，通过引入时间戳感知的多片段对齐机制，使MLLM能将问题与多个相关视频片段对齐，促进更全面的时间推理。为支持有效学习，设计了分阶段奖励策略，逐步引导模型实现时间对齐推理。大量实验表明，MUSEG在时间定位和时序敏感视频问答任务上显著优于现有方法，并在多种时间理解场景中具有良好泛化性。
原文摘要 · Abstract (English)
Video temporal understanding is crucial for multimodal large language models (MLLMs) to reason over events in videos. Despite recent advances in general video understanding, current MLLMs still struggle with fine-grained temporal reasoning. While reinforcement learning (RL) has been explored to address this issue recently, existing RL approaches remain limited in performance on time-sensitive tasks. In this work, we propose MUSEG, a novel RL-based method that enhances temporal understanding by introducing timestamp-aware multi-segment grounding. MUSEG enables MLLMs to align queries with multiple relevant video segments, promoting more comprehensive temporal reasoning. To facilitate effective learning, we design a customized RL training recipe with phased rewards that progressively guides the model toward temporally grounded reasoning. Extensive experiments on temporal grounding and time-sensitive video question answering (QA) tasks demonstrate that MUSEG significantly outperforms existing methods and generalizes well across diverse temporal understanding scenarios. View our project at https://github.com/THUNLP-MT/MUSEG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。