arXiv:2606.17730cs.CV2026-06被引 9

让虚拟世界能真正互动,支持捡东西开门等操作

ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

论文配图:ActWorld: From Explorable to Interactive World Model via Action-Aware Memory
图 1 · 摘自论文原文
  • 用分块自回归框架+动作感知记忆,实现边播放边交互
  • 在10万段带精细标注的互动视频上训练,交互准确率显著提升
  • 适合做游戏、机器人仿真等需要真实物体交互的场景

交互式世界模型旨在实时响应用户动作模拟环境动态。但现有模型的动作语义多局限于导航(如行走、转向),对场景中物体的交互(如捡盘子、开门、触发物理反应)支持不足,或仅限于游戏领域,或依赖提示生成完整视频。导致世界可视觉浏览却不可真正操作。本文提出ActWorld,一种扩展导航型生成器的交互式世界模型,在分块自回归框架下支持滚动过程中的物体交互。我们指出,导航与交互之间的差距源于两大瓶颈:一是数据瓶颈——缺乏带精确密集标注的人物-物体交互数据;二是记忆瓶颈——现有模型历史压缩存在近因偏差,丢弃了决定后续物体状态的事件过渡帧,造成“动作遗忘”。为此,我们构建了一个包含10万段互动视频的数据集,每段通过思维链推理标注分块描述。模型方面,引入分层动作感知记忆机制,按交互重要性路由历史压缩,并配备持久记忆库,保持事件更新和物体身份标记在长序列中的连续性。实验表明,ActWorld能在单一模型中同时支持灵活导航与丰富物体交互,显著提升交互保真度,且不牺牲视角控制能力。

原文摘要 · Abstract (English)

Interactive world models aim to simulate environment dynamics under real-time user actions. However, their action vocabulary is largely confined to navigation: most actions correspond to motion (e.g., walk, turn, look around), while interaction with objects in the scene (e.g., pick up plates, open doors, or trigger physical responses) is either absent, restricted to game domains, or relegated to prompt-to-full-video scenarios. The resulting worlds are visually explorable but not truly actionable. In this work, we present ActWorld, an interactive world model that extends prior navigation-centric generators to support mid-rollout object interaction within a chunk-autoregressive framework. We argue that the navigation-interaction gap stems from two bottlenecks. First, a data bottleneck: the lack of human-object interaction data with accurate, dense labels. Second, a memory bottleneck: recency-biased history compression in existing world models discards the event-transition frames that causally determine subsequent object states, leading to an action-forgetting pathology. On the data side, we construct a 100K interaction video dataset, each annotated with per-chunk captions via chain-of-thought reasoning. On the model side, we introduce a hierarchical action-aware memory design that routes history compression by interaction importance, complemented by a persistent memory bank that maintains event-update and object-identity tokens across long rollouts. Experiments show that ActWorld supports both flexible navigation and rich object interaction within a single model, substantially improving interaction fidelity over navigation-only baselines without sacrificing viewpoint control. Project page is available at https://interactwm.github.io/ActWorld.

世界模型物体交互记忆机制自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。