arXiv:2605.25333cs.CV2026-05被引 1

让视频生成模型像人一样记住中断后的状态变化。

Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution

论文配图:Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution
图 1 · 摘自论文原文
  • 用事件驱动数据与记忆中断增强,训练模型跨间隔回忆过去状态
  • 在STEVO-Bench上表现最优,能准确恢复被遮挡的动态变化
  • 适合需要长期依赖建模的视频生成、动作预测任务

视频世界模型应在无观测时持续演化状态,但现有生成器常在中断后冻结隐状态。这并非容量问题:预训练视频扩散变换器已有支持非局部检索的键值缓存机制,却很少被训练用于动态记忆。我们提出ReMind框架,通过面向记忆的数据、事件感知训练和缓存适配,激发动态记忆行为。基于100+类动态事件构建摄像头标注的混合训练数据,融合视觉语言模型筛选的真实视频、生成的高难度动态、合成摄像机循环及记忆中断增强样本。每段视频转为带保护锚点、退化区间和显式时间空隙的帧图。采用节点结构课程学习(包括节点删除、噪声记忆、前沿延续和参考缓存训练),强制模型在中断后仍能检索相关历史状态,而非依赖局部连续性。PM-RoPE是一种优雅的相机相位位置编码扩展,在单次注意力开销下实现时空检索,同时保留预训练路径。ReMind在STEVO-Bench和恢复任务中取得最佳综合得分。此外,通用图像到视频评估表明该课程避免灾难性遗忘。代码、数据与模型已开源。

原文摘要 · Abstract (English)

Video world models should maintain evolving states when evidence is unobserved, yet current generators often freeze hidden states upon interruption. This is not simply a capacity problem: pretrained video diffusion transformers already possess KV-cache mechanisms capable of non-local retrieval, but they are rarely trained to use them as dynamic memory. We introduce ReMind, a framework eliciting dynamic memory behavior via memory-oriented data, event-aware training, and cache adaptation. Organized around a taxonomy of 100+ dynamic events, we build a camera-annotated training mixture combining VLM-filtered real videos, generated hard dynamics, synthetic camera loops, and memory-interruption augmentations. Each clip is converted into a frame graph with protected anchors, degraded intervals, and explicit temporal gaps. A node-structured curriculum -- including node-drop, noisy memory, frontier continuation, and reference-cache training -- forces the model to retrieve relevant past states across interruptions rather than relying solely on local continuity. PM-RoPE, an elegant camera-phase RoPE extension, unlocks spatiotemporal retrieval at a single-attention cost while preserving pretrained pathways. ReMind achieves the best overall scores on STEVO-Bench and recovery tasks. Furthermore, general image-to-video evaluations confirm this curriculum avoids catastrophic forgetting. We have released our code, data, and models on our project page \href{https://remind-applied.github.io/}{https://remind-applied.github.io/}.

视频生成动态记忆扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。