arXiv:2608.07409cs.CV2026-08中稿 · ICML

统一视觉世界建模框架,图像与视频任务共用一个模型。

UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling

论文配图:UniJEPA: A Unified Joint-Embedding Predictive Architecture for Task-Agnostic Visual World Modeling
图 1 · 摘自论文原文
  • 用单一损失函数联合学习光照变换与视频时序预测。
  • 无需预训练或复杂正则,端到端训练且防崩溃。
  • 支持零样本规划,推理速度比生成模型快数十倍。

联合嵌入预测架构(JEPAs)已成为自监督学习紧凑潜在空间中世界模型的原理性框架,但现有方法分散:部分预测单张图像的掩码区域(I-JEPA),部分学习预测全局光度变换(图像世界模型),视频级JEPAs则预测未来时序状态并用于动作条件规划(V-JEPA~2、DINO-World、DINO-WM)。这些目标被当作独立方案,使用不同编码器、预测器和防崩溃正则,阻碍了图像与视频级建模的统一。本文提出UniJEPA,一种统一的JEPA,在共享潜在空间中联合学习光度预测(图像级变换)与时序预测(视频级下一状态动态)。单一端到端目标由下一嵌入预测损失与高斯正则组成,可证明防崩溃,直接从原始像素训练,无需EMA、停止梯度或预训练编码器。我们发现同一潜在空间支持可控抽象:光度预测学习不变结构,时序预测学习等变动态。在离线轨迹上进行动作条件后训练后,可通过将目标特征作为预测目标实现零样本规划。在图像、视频与控制基准测试中,UniJEPA性能媲美甚至超越专用JEPAs,仅需一个损失超参数,且规划速度比生成式世界模型快数十倍,精度相当。

原文摘要 · Abstract (English)

Joint-Embedding Predictive Architectures (JEPAs) have emerged as a principled framework for self-supervised learning of world models in compact latent spaces, yet existing methods are fragmented: some predict masked parts of a single image in latent space (I-JEPA), others learn to predict global photometric transformations (Image World Models), while video-scale JEPAs predict future temporal states and are post-trained for action-conditioned planning (V-JEPA~2, DINO-World, DINO-WM). These objectives are treated as distinct recipes with separate encoders, predictors, and anti-collapse regularizers, hindering a single model from unifying image-level and video-level world modeling. We present UniJEPA, a unified JEPA that jointly learns photometric prediction (image-level transformations) and temporal prediction (video-level next-state dynamics) in one shared latent space. A single end-to-end objective, composed of a next-embedding prediction loss and a Gaussian regularizer, yields a provably anti-collapse encoder-predictor pair trainable from raw pixels without EMA, stop-gradient, or pre-trained encoders. We show that the same latent space supports controllable abstraction: photometric prediction learns invariant structure while temporal prediction learns equivariant dynamics. After action-conditioned post-training on offline trajectories, UniJEPA enables zero-shot planning by treating goal features as prediction targets. On image, video, and control benchmarks, UniJEPA matches or surpasses task-specific JEPAs while requiring a single loss hyperparameter, and plans up to tens of times faster than generative world models at comparable accuracy.

自监督学习世界模型统一架构零样本规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。