arXiv:2602.03604cs.CVcs.AI2026-02被引 11

轻量级库实现能量基联合嵌入预测架构,支持从图像到动作控制的世界模型学习。

A Lightweight Library for Energy-Based Joint-Embedding Predictive Architectures

  • 在表示空间中预测,避免生成建模缺陷,保留语义特征。
  • 在CIFAR-10上达91%探测准确率,视频多步预测成功,导航任务规划成功率97%。
  • 单卡几小时可训练,适合研究与教学,代码开源易复现。

我们提出EB-JEPA,一个开源库,用于通过联合嵌入预测架构(JEPAs)学习表征和世界模型。JEPAs在表示空间而非像素空间中进行预测,避免了生成建模的缺陷,同时捕捉适用于下游任务的语义特征。该库提供模块化、自包含的实现,展示图像级自监督学习的表征技术如何扩展至视频建模(引入时间动态复杂性),最终应用于动作条件世界模型(需预测控制输入的影响)。每个示例均设计为单GPU数小时内完成训练,使基于能量的自监督学习更易用于研究与教育。我们在CIFAR-10上进行组件消融,探测表征达91%准确率,表明模型学习到有效特征。视频部分以Moving MNIST为例展示多步预测能力,验证原理可扩展至时序建模。最后,展示这些表征驱动动作条件世界模型,在Two Rooms导航任务中达到97%规划成功率。全面消融分析揭示各正则化组件对防止表征坍塌至关重要。代码已开源:https://github.com/facebookresearch/eb_jepa。

原文摘要 · Abstract (English)

We present EB-JEPA, an open-source library for learning representations and world models using Joint-Embedding Predictive Architectures (JEPAs). JEPAs learn to predict in representation space rather than pixel space, avoiding the pitfalls of generative modeling while capturing semantically meaningful features suitable for downstream tasks. Our library provides modular, self-contained implementations that illustrate how representation learning techniques developed for image-level self-supervised learning can transfer to video, where temporal dynamics add complexity, and ultimately to action-conditioned world models, where the model must additionally learn to predict the effects of control inputs. Each example is designed for single-GPU training within a few hours, making energy-based self-supervised learning accessible for research and education. We provide ablations of JEA components on CIFAR-10. Probing these representations yields 91% accuracy, indicating that the model learns useful features. Extending to video, we include a multi-step prediction example on Moving MNIST that demonstrates how the same principles scale to temporal modeling. Finally, we show how these representations can drive action-conditioned world models, achieving a 97% planning success rate on the Two Rooms navigation task. Comprehensive ablations reveal the critical importance of each regularization component for preventing representation collapse. Code is available at https://github.com/facebookresearch/eb_jepa.

表征学习世界模型自监督轻量库

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。