arXiv:2603.25716cs.CVcs.AI2026-03被引 13

让视频模型学会追踪躲藏的动态物体,保持运动连续性。

Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

  • 设计双模式记忆机制,分别处理静态背景和动态物体
  • 在59,000段视频上实现隐藏物体身份与动作的精准保留
  • 适合做视频生成、物理模拟、机器人感知的研究者

视频世界模型在模拟物理世界方面潜力巨大,但现有记忆机制多将环境视为静态场景。当动态物体隐没于视野之外再出现时,当前方法常导致物体冻结、失真或消失。为此,我们提出混合记忆(Hybrid Memory)新范式,要求模型同时作为静态背景的精确档案员和动态物体的警觉追踪者,确保物体在视野外期间仍保持运动连续性。为推动该方向研究,我们构建了首个专注于混合记忆的大规模视频数据集HM-World,包含59,000段高保真视频片段,解耦相机与物体轨迹,涵盖17个场景、49种不同物体,并精心设计退出-进入事件以严格评估混合一致性。此外,我们提出专用于该任务的记忆架构HyDRA,将记忆压缩为令牌,并采用时空相关性驱动的检索机制,通过选择性关注相关运动线索,有效保留隐藏物体的身份与运动特征。在HM-World上的大量实验表明,本方法在动态物体一致性与整体生成质量上均显著优于现有最先进方法。代码已公开于https://github.com/H-EmbodVis/HyDRA。

原文摘要 · Abstract (English)

Video world models have shown immense potential in simulating the physical world, yet existing memory mechanisms primarily treat environments as static canvases. When dynamic subjects hide out of sight and later re-emerge, current methods often struggle, leading to frozen, distorted, or vanishing subjects. To address this, we introduce Hybrid Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals. To facilitate research in this direction, we construct HM-World, the first large-scale video dataset dedicated to hybrid memory. It features 59K high-fidelity clips with decoupled camera and subject trajectories, encompassing 17 diverse scenes, 49 distinct subjects, and meticulously designed exit-entry events to rigorously evaluate hybrid coherence. Furthermore, we propose HyDRA, a specialized memory architecture that compresses memory into tokens and utilizes a spatiotemporal relevance-driven retrieval mechanism. By selectively attending to relevant motion cues, HyDRA effectively preserves the identity and motion of hidden subjects. Extensive experiments on HM-World demonstrate that our method significantly outperforms state-of-the-art approaches in both dynamic subject consistency and overall generation quality. Code is publicly available at https://github.com/H-EmbodVis/HyDRA.

视频生成记忆机制动态追踪

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。