让视频能自动预测并生成下一步动作,像教人打结一样直观。
Video-as-Answer: Predict and Generate Next Video Event with Joint-GRPO
- 用联合强化学习让视觉语言模型和视频生成模型协同工作
- 在10万条数据上训练,视频生成与描述更一致且准确
- 适合需要动手教学或创意演示的场景
尽管语言模型已在众多实际应用中展现价值,视频生成仍主要局限于娱乐领域。鉴于视频能直观展现文字难以表达的物理世界信息(如仅靠文字教人系领带),我们提出将视频作为下一事件预测的新答案模态,即视频下一事件预测(VNEP)。与传统以文本回答下一事件不同,VNEP要求生成动态视频响应,使答案更直观、可定制,适用于操作学习与创意探索。但现有模型难以胜任,因需理解多模态输入、指令引导推理,并生成视觉与语义一致的视频。为此,我们提出VANS模型,利用强化学习对齐视觉-语言模型(VLM)与视频扩散模型(VDM),核心为提出的联合GRPO方法,使其协同工作。两者共享奖励信号:优化VLM生成易可视化且准确的描述,同时引导VDM生成符合描述与输入视觉上下文的视频。为支持训练,我们构建了专用于VNEP任务的VANS-Data-100K数据集。在程序性与预测性基准测试中,VANS在视频事件预测与可视化方面均达到当前最优性能。代码已开源。
原文摘要 · Abstract (English)
While language models have become impactful in many real-world applications, video generation remains largely confined to entertainment. Motivated by video's inherent capacity to demonstrate physical-world information that is difficult to convey through language alone (e.g., imagine teaching someone to tie a tie using only text), we identify an underutilized opportunity to extend video as a new answer modality for Next-Event Prediction (NEP), formalized as Video-Next-Event Prediction (VNEP). While the established NEP task takes a video with a procedural or predictive question as input to predict the next event in text, VNEP requires dynamic video responses. This shift from telling to showing unlocks more intuitive and customized answers for procedural learning and creative exploration. However, this task remains challenging for existing models, as it demands an understanding of multimodal input, instruction-conditioned reasoning, and the generation of video with visual and semantic consistency. To address this, we introduce VANS, a model that leverages reinforcement learning to align a Vision-Language Model (VLM) with a Video Diffusion Model (VDM) for VNEP. The core of VANS is our proposed Joint-GRPO that orchestrates the VLM and VDM to function as a unit. Driven by a shared reward on their respective output, it optimizes the VLM to produce captions that are both accurate and friendly to visualize, while guiding the VDM to generate videos that are faithful to these captions and the input visual context. To enable this learning, we craft VANS-Data-100K, a dedicated dataset for the VNEP task. Experiments on procedural and predictive benchmarks demonstrate that VANS achieves state-of-the-art performance in both video event prediction and visualization. Codes are released in https://github.com/KlingTeam/VANS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。