arXiv:2608.20974cs.CVcs.AI2026-08

WA-JEPA改进视频预测框架,专为自动驾驶规划设计。

WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving

论文配图:WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving
图 1 · 摘自论文原文
  • 用未来掩码预训练替代随机掩码,增强对未来状态的建模能力。
  • 通过条件流匹配生成合理未来潜变量,提升规划可用性。
  • 联合预测未来场景与车辆轨迹,实现动作监督直接优化世界表征。

视频联合嵌入预测架构(V-JEPA)通过自监督潜变量预测学习强大的时空表示,但其基于随机掩码补全和确定性回归的机制,难以满足自动驾驶规划对目标导向未来预测的需求。为此,本文重新思考V-JEPA范式,提出WA-JEPA——一种面向自动驾驶规划的原生世界-动作模型。该模型采用混合未来掩码预训练,从观测上下文中推断未来潜变量;将未来预测重构为潜变量未来的条件流匹配,显著提升生成合理未来潜变量的能力;并引入联合未来-动作预测器,在统一时空潜空间中共同去噪未来场景标记与自身轨迹,使动作监督可直接塑造规划相关世界表示。在nuPlan视频上预训练并在NAVSIM上微调后,WA-JEPA在NAVSIM-v2上达到91.7 EPDMS,优于最强端到端及世界-动作基线1.6和1.3 EPDMS;无需针对HUGSIM微调,即在闭环HUGSIM基准上取得0.4462最优HD-Score。结果验证了原生V-JEPA世界-动作建模在自动驾驶规划中的有效性与可扩展性。代码已开源。

原文摘要 · Abstract (English)

Video Joint Embedding Predictive Architecture (V-JEPA) learns powerful spatiotemporal representations from video through self-supervised latent feature prediction. However, V-JEPA is built around random-mask completion and deterministic regression, making it fundamentally ill-suited for autonomous driving planning that demands future-directed prediction tightly coupled with action. To address this, we rethink the V-JEPA paradigm and present WA-JEPA, a V-JEPA-native world-action model designed for autonomous driving planning. Instead of random spatiotemporal masking, WA-JEPA employs hybrid future-masked pre-training, where the model infers future latents from observed context. Departing from deterministic regression, we recast future prediction as conditional flow matching over latent futures, which substantially improves the model's ability to generate plausible future latents for downstream planning. Finally, a joint future-action predictor is proposed to denoise future scene tokens and ego trajectories together in a unified spatiotemporal latent space, allowing action supervision to directly shape planning-relevant world representations. Pre-trained on nuPlan videos and fine-tuned on NAVSIM, WA-JEPA reaches 91.7 EPDMS on NAVSIM-v2, surpassing the strongest end-to-end and world-action baselines by 1.6 and 1.3 EPDMS, and, without HUGSIM-specific fine-tuning, attains the best HD-Score of 0.4462 on the closed-loop HUGSIM benchmark under the same evaluation protocol. These results validate V-JEPA-native world-action modeling as a powerful and scalable paradigm for autonomous driving planning. Code is available at https://github.com/AFARI-Research/WA-JEPA.

自动驾驶视频预测潜变量建模规划

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。