让视觉-语言-动作模型学会预测未来世界,提升智能体规划能力。
X-Foresight: A Joint Vision-Action Causal Forecasting Network via Predictive World Modeling

- 用分块自回归策略预测语义相隔的视频片段,避免简单重复。
- 在10秒内预测多步动作,规划成功率超基线32%以上。
- 适合需要长期推理与安全决策的自动驾驶等场景。
物理世界知识主要存在于视频中。赋予视觉-语言-动作(VLA)模型此类知识是实现安全、泛化性规划的基础。预测性世界建模通过从历史观测预测未来视频,使VLA内化物理动态与长时因果关系。然而,直接的下一帧预测面临两大挑战:1)视频帧令牌熵值低且冗余,导致预测退化为平凡外推;2)建模存在时间困境:密集预测可捕捉瞬时动态,但难以高效建模长时因果。为此,我们提出X-Foresight,一种直接集成于VLA架构的预测性世界模型,联合学习世界建模与实时动作控制。其核心采用长时程分块自回归策略:通过预测语义相距较远的视频块而非相邻帧,避免平凡外推;同时保留块内密集帧以捕捉瞬时动态,块间稀疏跳变以建模长时因果。引入课程学习调度逐步延长预测时长,稳定训练过程。为有效捕捉长时因果,提出时间重要性采样,聚焦于由自身运动和行为信号识别出的安全关键片段。进一步将高保真生成任务交由基于扩散的多视角渲染器处理,提升图像真实感。大量实验表明,X-Foresight在规划性能上显著优于现有VLA基线,同时保持强生成保真度,建立了一个以世界知识驱动的自主系统新范式。
原文摘要 · Abstract (English)
Physical world knowledge resides mainly in videos. Equipping Vision-Language-Action (VLA) models with such knowledge is fundamental for safe and generalizable planning. Predictive world modeling enables VLA to internalize physical dynamics and long-term causality by predicting future video from past observations. However, naive next-frame prediction faces two challenges: 1) unlike semantically distinct text tokens, video tokens are low-entropy and redundant, causing prediction to degenerate into trivial extrapolation. 2) world modeling poses a temporal dilemma: dense prediction captures instantaneous dynamics, but cannot efficiently model long-horizon causality. To learn world knowledge effectively, we introduce X-Foresight, a predictive world model integrated directly into the VLA architecture to jointly learn world modeling and real-time action control. At its core lies a long-horizon chunk-wise auto-regressive strategy that addresses both challenges: by predicting semantically distant chunks rather than adjacent frames, it escapes trivial extrapolation, while preserving dense intra-chunk frames for instantaneous dynamics and sparse inter-chunk transitions for long-term causality. A curriculum learning schedule progressively extends prediction horizons and stabilizes long-horizon training. To capture long-term causality effectively, we present temporal importance sampling, which concentrates supervision on safety-critical chunks identified by ego-motion and behavioral signals. We further delegate photorealistic synthesis to a diffusion-based multi-view renderer, improving photorealistic appearance. Comprehensive experiments demonstrate that X-Foresight significantly outperforms VLA baselines in planning performance while maintaining strong generative fidelity, establishing a robust paradigm for world-knowledge-driven autonomous systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。