输入图片和物理条件，生成符合真实物理的动态视频
PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation
- 结合物理仿真与数据驱动生成，模拟图像空间中的真实运动
- 在多个指标上优于现有方法，用户研究验证其真实感更强
- 适合需要可控物理动画的场景，如交互式内容创作
我们提出PhysGen，一种新颖的图像到视频生成方法，可将单张图像与输入条件（如作用于图像中物体的力和力矩）转换为逼真、物理合理且时间一致的视频。核心思想是将基于模型的物理仿真与数据驱动的视频生成相结合，实现图像空间中的合理动力学。系统包含三个核心组件：(i) 图像理解模块，有效捕捉图像中的几何、材质和物理参数；(ii) 图像空间动力学仿真模型，利用刚体物理和推断参数模拟真实行为；(iii) 基于图像的渲染与优化模块，借助生成式视频扩散模型生成包含模拟运动的逼真视频画面。生成的视频在物理和外观上均具真实性，且具备精确可控性，通过定量比较和全面用户研究验证了其优越性能。PhysGen生成的视频可用于多种下游应用，如将图像转化为真实动画，或让用户与图像互动以生成各种动态效果。
原文摘要 · Abstract (English)
We present PhysGen, a novel image-to-video generation method that converts a single image and an input condition (e.g., force and torque applied to an object in the image) to produce a realistic, physically plausible, and temporally consistent video. Our key insight is to integrate model-based physical simulation with a data-driven video generation process, enabling plausible image-space dynamics. At the heart of our system are three core components: (i) an image understanding module that effectively captures the geometry, materials, and physical parameters of the image; (ii) an image-space dynamics simulation model that utilizes rigid-body physics and inferred parameters to simulate realistic behaviors; and (iii) an image-based rendering and refinement module that leverages generative video diffusion to produce realistic video footage featuring the simulated motion. The resulting videos are realistic in both physics and appearance and are even precisely controllable, showcasing superior results over existing data-driven image-to-video generation works through quantitative comparison and comprehensive user study. PhysGen's resulting videos can be used for various downstream applications, such as turning an image into a realistic animation or allowing users to interact with the image and create various dynamics. Project page: https://stevenlsw.github.io/physgen/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。