让视频生成符合物理规律，解决物体运动不真实问题
Physical Simulator In-the-Loop Video Generation
- 将物理模拟器嵌入扩散模型，生成符合重力等规律的运动轨迹
- 在多个数据集上显著提升物理一致性，同时保持视觉质量
- 适合需要真实物理行为的视频生成场景，如影视特效、仿真
基于扩散的视频生成虽已实现高度视觉真实感，但仍难以遵循基本物理规律，如重力、惯性与碰撞。生成物体常出现帧间运动不一致、动力学不合理或违反物理约束等问题，影响视频的真实性和可靠性。为此，我们提出物理模拟器内循环视频生成（PSIVG）框架，将物理模拟器融入视频扩散过程。从预训练扩散模型生成的模板视频出发，PSIVG重建4D场景与前景物体网格，将其初始化至物理模拟器中，生成物理一致的运动轨迹，并以此引导视频生成器实现时空上一致的物理运动。为进一步提升物体移动时的纹理一致性，提出一种测试时纹理一致性优化（TTCO）技术，根据模拟器中的像素对应关系动态调整文本与特征嵌入。大量实验表明，PSIVG生成的视频更符合现实物理规律，同时保持视觉质量与多样性。
原文摘要 · Abstract (English)
Recent advances in diffusion-based video generation have achieved remarkable visual realism but still struggle to obey basic physical laws such as gravity, inertia, and collision. Generated objects often move inconsistently across frames, exhibit implausible dynamics, or violate physical constraints, limiting the realism and reliability of AI-generated videos. We address this gap by introducing Physical Simulator In-the-loop Video Generation (PSIVG), a novel framework that integrates a physical simulator into the video diffusion process. Starting from a template video generated by a pre-trained diffusion model, PSIVG reconstructs the 4D scene and foreground object meshes, initializes them within a physical simulator, and generates physically consistent trajectories. These simulated trajectories are then used to guide the video generator toward spatio-temporally physically coherent motion. To further improve texture consistency during object movement, we propose a Test-Time Texture Consistency Optimization (TTCO) technique that adapts text and feature embeddings based on pixel correspondences from the simulator. Comprehensive experiments demonstrate that PSIVG produces videos that better adhere to real-world physics while preserving visual quality and diversity. Project Page: https://vcai.mpi-inf.mpg.de/projects/PSIVG/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。