用傅里叶头让世界模型学得更物理,提升动态环境下的规划成功率。
Spectral-Target Physical Latent Structuring for JEPA-Style World Models

- 训练时加轻量傅里叶辅助头,强制潜在空间具备物理结构。
- 动态环境中规划成功率显著提升,低数据下效果更明显。
- 不增加推理开销,适配任意环境,改善表征物理性。
潜在世界模型通过在潜在空间而非像素空间中进行预测与规划,日益受到关注。近期架构如LeWorldModel(LeWM)采用SIGReg等正则化技术联合训练编码器与预测器,防止表征坍缩。然而,即便如此,我们发现了一种新失败模式—— extit{物理表征懒惰},尤其在高度动态环境中表现明显:学习到的潜在状态虽未坍缩,却无法体现关键物理属性,导致下游规划普遍失败。为此,我们提出训练时引入轻量级“傅里叶辅助头”作为附加监督,以无额外推理成本的方式强制潜在空间具备物理感知结构,且可泛化至任意环境。实验表明,该方法显著提升了动态环境中的规划成功率,基线模型(LeWM)在此类场景中表现出物理表征懒惰;在其他环境亦有小幅改进,即使基线未出现懒惰现象。进一步观察发现,规划性能提升伴随潜在空间与关键物理属性相关性增强,表明本方法有效构建了具有物理意义的潜在表征,并可能为规划带来优势。在低数据条件下,辅助监督影响尤为显著,大幅提升成功率。这些结果支持使用傅里叶辅助头提升整体成功率与数据效率,同时避免潜在世界模型中的表征懒惰问题。
原文摘要 · Abstract (English)
Latent world models have become increasingly popular as a method to predict and plan in latent space rather than pixel space. Recent architectures, such as LeWorldModel (LeWM), jointly train the encoder and predictor using regularization techniques like SIGReg to prevent representation collapse. Even with such regularization preventing representation collapse, we identify a new world model failure mode of \textit{physical representation laziness}, particularly noted in highly dynamic environments. For these lazy cases, the learned latent states do not collapse but nonetheless fail to represent key physical properties, causing ubiquitous downstream planning failure. To resolve this issue, we propose training-time auxiliary supervision with a lightweight "Fourier auxiliary head", which enforces physically-informed structuring of the latent space with no additional inference-time cost and can be generalized to any environment. Experimentally, we show that the auxiliary head substantially improves planning success rates in dynamic environments where the baseline LeWM exhibits physical representation laziness. It also leads to modest improvements in other environments, even when the baseline does not exhibit physical representation laziness. We further observe superior planning performance being accompanied by higher latent space correlations with key physical properties, indicating both the ability of our method to physically structure latent states and the potential planning-side benefit to the learned representation being physically structured. We also see in low-data regimes, auxiliary supervision is particularly impactful in increasing success rate. These findings support the use of our Fourier auxiliary head method to improve both overall success rate and data efficiency, while avoiding representation laziness in latent world models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。