通过物体级隐变量掩码,让世界模型学会因果推理。
Causal-JEPA: Learning World Models through Object-Level Latent Masking
- 用物体级隐变量掩码训练,强制模型从上下文推断被遮挡物体。
- 在反事实推理任务上比基线提升约20%准确率。
- 只需1%的特征就能实现高效规划,适合机器人控制场景。
世界模型需要强大的关系理解能力以支持预测、推理和控制。尽管以物体为中心的表示提供了有用抽象,但不足以捕捉依赖交互的动力学。为此,我们提出C-JEPA,一种简单且灵活的物体中心世界模型,将图像块的掩码联合嵌入预测扩展至物体级表示。通过掩码物体级隐变量,并要求每个被掩码物体状态从周围上下文中推断,C-JEPA在训练中引入结构化部分可观测性,形成类似反事实的预测任务,从而抑制捷径解决方案,使交互依赖预测成为学习目标的必要条件。实验表明,C-JEPA在视觉问答任务中带来一致性能提升,反事实推理准确率相比无物体级掩码的同架构模型提升约20%。在智能体控制任务中,仅需补丁基世界模型1%的隐变量输入特征即可实现相当性能,显著提升规划效率。最后,我们提供形式化分析,证明物体级掩码通过控制可观测性引入有益归纳偏置。代码已开源:https://github.com/galilai-group/cjepa。
原文摘要 · Abstract (English)
World models require robust relational understanding to support prediction, reasoning, and control. While object-centric representations provide a useful abstraction, they are not sufficient to capture interaction-dependent dynamics. We therefore propose C-JEPA, a simple and flexible object-centric world model that extends masked joint embedding prediction from image patches to object-centric representations. By masking object-level latents and requiring each masked object state to be inferred from the surrounding context, C-JEPA imposes structured partial observability during training, creating counterfactual-like prediction queries that discourage shortcut solutions and make interaction-dependent prediction necessary under the learning objective. Empirically, C-JEPA leads to consistent gains in visual question answering, with an absolute improvement of about 20% in counterfactual reasoning over the same architecture without object-level masking. On agent control tasks, C-JEPA enables substantially more efficient planning by using only 1% of the total latent input features required by patch-based world models, while achieving comparable performance. Finally, we provide a formal analysis demonstrating that object-level masking induces useful inductive bias by controlling observability. Our code is available at https://github.com/galilai-group/cjepa.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。