提出新基准与方法,让视频生成能准确追踪世界状态变化。
Do Video Generators Track the World Across Segments? A Benchmark and Method for World-State Reasoning in Video Continuation

- 设计实体状态表示,显式维护并更新视频世界状态
- 在三类状态任务中,综合得分从45.2提升至69.3
- 适合需要连贯叙事的长视频生成场景
视频生成通过拼接短片段或自回归扩展来构建长视频,依赖历史帧、关键帧或缓存特征等记忆。然而当前模型难以将这些可见证据转化为可靠的‘世界状态’接口:即前序动作后世界的真实状态,以及如何随新提示改变。过去帧虽为有效历史,但未必反映下一阶段所需状态;部分状态需从遮挡或隐含变化中推断,而非直接复制。为此,我们提出Statebench基准,评估三种状态类型下的延续能力:可观察状态、遮挡过程状态、复杂过渡状态。进一步提出Stateagent,显式维护实体状态,根据新提示更新,并以未来终帧为锚点生成下一段视频。实验表明,该方法使综合状态得分(SCS-All)从45.2提升至69.3,显著改善可控视频延续,且在分钟级故事生成中表现优异。代码已开源。
原文摘要 · Abstract (English)
Video generators build long videos by composing shorter parts, either by generating segments one after another or by autoregressively extending chunks. Each new part usually depends on memories of historical observations, such as recent frames, selected key frames, memory banks, or cached features. These memories preserve visible evidence from the past, but current generators do not reliably turn such evidence into a world-state interface: what holds in the video world after previous actions and how it should change under the next prompt. A past frame remains valid history, but it may not describe the state needed by the next segment; some states must instead be inferred from occluded or implicit changes rather than copied from a directly observed frame. This creates a simple but overlooked question for video continuation: given a previous video, its prompt, and a new prompt, can a model generate a continuation that reflects the state determined by both the historical video and the new prompt? To answer this question, we introduce Statebench, a benchmark that targets this gap by testing continuations over three state categories: past-visible states, occluded-process states, and complex-transition states. We further propose Stateagent, which explicitly maintains an entity-state representation, updates it under the new prompt, grounds the predicted post-action state as a future end frame, and renders the next video. Experiments show that our method improves controlled video continuation by raising the all-case state score (SCS-All) from 45.2 to 69.3, and also benefits story generation at the one-minute scale. Code is avaliable at https://github.com/AMAP-ML/StateAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。