arXiv:2506.06006cs.CVcs.AI2025-06ACL被引 3

用视觉语言模型预测未来图像,靠逆动力学来引导。

Can VLMs Predict Future States? Bootstrapping World Models from Inverse Dynamics

  • 让模型先学会反推动作描述,再用它生成未来图像。
  • 在Aurora-Bench上比顶尖图像编辑模型高7%~13%。
  • 适合做视频生成、动作控制的开发者参考。

统一的视觉语言模型(VLM)能否实现前向动力学预测(FDP),即根据前一帧观测和语言形式的动作指令预测未来的图像状态?我们发现,直接使用VLM生成物理上合理的帧间过渡存在困难。然而,我们识别出一种关键的多模态对齐不对称性:微调VLM进行逆动力学预测(IDP)——即为两帧之间的动作生成描述——远比学习FDP容易。进而,可利用IDP通过两种策略引导FDP:一是弱监督学习,用合成数据扩充训练集;二是推理时验证,利用IDP为多个FDP候选样本打分以指导搜索。我们在Aurora-Bench上的动作中心图像编辑任务中评估这两种策略。尽管仍是通用模型,最佳方案性能已媲美最先进图像编辑模型,在GPT4o评分下提升7%至13%,且在所有子集的人类平均评价中表现最优。

原文摘要 · Abstract (English)

Can unified vision-language models (VLMs) perform forward dynamics prediction (FDP), i.e., predicting the future state (in image form) given the previous observation and an action (in language form)? We find that VLMs struggle to generate physically plausible transitions between frames from instructions. Nevertheless, we identify a crucial asymmetry in multimodal grounding: fine-tuning a VLM to learn inverse dynamics prediction (IDP)-effectively captioning the action between frames-is significantly easier than learning FDP. In turn, IDP can be used to bootstrap FDP through two main strategies: 1) weakly supervised learning from synthetic data and 2) inference time verification. Firstly, IDP can annotate actions for unlabelled pairs of video frame observations to expand the training data scale for FDP. Secondly, IDP can assign rewards to multiple samples of FDP to score them, effectively guiding search at inference time. We evaluate the FDP resulting from both strategies through the task of action-centric image editing on Aurora-Bench with two families of VLMs. Despite remaining general-purpose, our best model achieves a performance competitive with state-of-the-art image editing models, improving on them by a margin between 7% and 13% according to GPT4o-as-judge, and achieving the best average human evaluation across all subsets of Aurora-Bench.

视觉语言模型动态预测视频生成逆动力学

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。