用未来视频片段训练模型预测事件，提升多模态大模型的时间推理能力。
Fostering Video Reasoning via Next-Event Prediction
- 通过预测未来事件摘要，引导模型进行时间推理。
- 在3.3万段自动提取的视频上验证，显著提升时序理解能力。
- 适合研究视频理解、时序推理的学者与开发者参考。
在多模态大模型中，时间推理能力仍面临挑战。现有任务如视频问答依赖人工标注或强模型，视频字幕则混淆时空信息。为此，本文提出下一事件预测（Next-Event Prediction, NEP），将视频分为过去与未来帧：模型基于过去帧预测未来事件摘要，从而激励其进行时间推理。为支持该任务，我们构建了包含33,000段自动提取视频的V1-33K数据集，覆盖多样真实场景。进一步探索多种指令微调策略对时序推理的影响。为评估进展，提出FutureBench基准，用于衡量对未见未来事件的连贯预测能力。实验表明，NEP是一种可扩展且有效的训练范式，能有效促进多模态大模型的时间推理能力。
原文摘要 · Abstract (English)
Next-token prediction serves as the foundational learning task enabling reasoning in LLMs. But what should the learning task be when aiming to equip MLLMs with temporal reasoning capabilities over video inputs? Existing tasks such as video question answering often rely on annotations from humans or much stronger MLLMs, while video captioning tends to entangle temporal reasoning with spatial information. To address this gap, we propose next-event prediction (NEP), a learning task that harnesses future video segments as a rich, self-supervised signal to foster temporal reasoning. We segment each video into past and future frames: the MLLM takes the past frames as input and predicts a summary of events derived from the future frames, thereby encouraging the model to reason temporally in order to complete the task. To support this task, we curate V1-33K, a dataset comprising 33,000 automatically extracted video segments spanning diverse real-world scenarios. We further explore a range of video instruction-tuning strategies to study their effects on temporal reasoning. To evaluate progress, we introduce FutureBench to assess coherence in predicting unseen future events. Experiments validate that NEP offers a scalable and effective training paradigm for fostering temporal reasoning in MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。