让视频模型学会物理规律，能预测没见过的运动轨迹。
Learning How the World Evolves: Extrapolative Video World Models via Latent Dynamics Reasoning
- 用潜空间动力学推理，只学高阶残差，模拟物体运动规律。
- 在五类物理任务上，外推误差比基线小20倍以上，参数少26倍。
- 训练时只看红球左移，也能正确预测蓝方块右移，泛化能力强。
世界演化的规律体现在其动力学上，即运动法则。然而主流视频扩散模型主要拟合像素，未建模像素随时间的变化过程，导致生成画面虽逼真但可能违背物理规律。为从像素中纯学习动力学，我们提出潜空间动力学推理（LDR）。LDR将潜变量转移建模为显式的运动积分，仅回归三阶及以上残差以驱动演化；为更好外推，其在结构化潜空间而非密集卷积特征上执行积分。基于PhyWorld基准，在涵盖匀速、抛物线、碰撞、弹跳和逼近等五类任务的白盒物理环境中，重点评估分布外场景下的表现。结果表明，LDR的内外分布误差差距比视频扩散基线小20倍以上，且在256²分辨率下参数量减少26倍、推理速度提升143倍。即使训练仅包含左移红球，也能准确预测右移蓝方块的运动。据我们所知，这是首个能在训练分布外实现动力学有效外推的视频世界模型。
原文摘要 · Abstract (English)
The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20$\times$ smaller than the video diffusion baseline's, under both single- and joint-task training at 256$^2$ resolution, while using 26$\times$ fewer parameters and running 143$\times$ faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。