用大模型生成可解释的视频未来，让机器像人一样推理动作演化。
Programmatic Video Prediction Using Large Language Models
- 用大模型将视频帧转为可读程序，描述每帧状态和变化规律。
- 在物理仿真和小车平衡任务中，预测精度超越现有方法。
- 支持反事实推演，适合需要透明决策的自动驾驶等场景。
估计真实世界过程的动力学模型对预测和准备未来结果至关重要。在视频监控、机器人、自动驾驶等应用中，该目标要求基于几帧视频提供视觉上下文后，合成合理的视觉未来。为此，我们提出 ProgGen，通过利用大（视觉）语言模型（LLM/VLM）的归纳偏置，将视频动态表示为一组神经符号式、人类可读的状态序列（每帧一个状态），实现视频帧预测。具体而言，ProgGen 利用 LLM/VLM 生成程序：(i) 根据视觉上下文（即视频帧）估计视频状态；(ii) 通过估算状态转移动态预测未来时间步的状态；(iii) 将预测状态渲染为视觉 RGB 帧。实证评估表明，所提方法在两个挑战性环境（i）PhyWorld （ii）Cart Pole 中均优于现有技术。此外，ProgGen 支持反事实推理与可解释视频生成，证明其在视频生成任务中的有效性与泛化能力。
原文摘要 · Abstract (English)
The task of estimating the world model describing the dynamics of a real world process assumes immense importance for anticipating and preparing for future outcomes. For applications such as video surveillance, robotics applications, autonomous driving, etc. this objective entails synthesizing plausible visual futures, given a few frames of a video to set the visual context. Towards this end, we propose ProgGen, which undertakes the task of video frame prediction by representing the dynamics of the video using a set of neuro-symbolic, human-interpretable set of states (one per frame) by leveraging the inductive biases of Large (Vision) Language Models (LLM/VLM). In particular, ProgGen utilizes LLM/VLM to synthesize programs: (i) to estimate the states of the video, given the visual context (i.e. the frames); (ii) to predict the states corresponding to future time steps by estimating the transition dynamics; (iii) to render the predicted states as visual RGB-frames. Empirical evaluations reveal that our proposed method outperforms competing techniques at the task of video frame prediction in two challenging environments: (i) PhyWorld (ii) Cart Pole. Additionally, ProgGen permits counter-factual reasoning and interpretable video generation attesting to its effectiveness and generalizability for video generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。