让视频生成更符合物理规律,仅靠一次采样修正即可提升真实感。
Off-Manifold Refinement: Guiding Video Generators with a Frozen World Model

- 在采样过程中注入冻结世界模型的反馈,实时修正轨迹。
- 在400个提示上将物理合理性指标提升5.0个百分点至52.0%。
- 推理成本仅增加1.71倍,远低于多轮搜索或重训练方案。
现代视频生成器常忽略物理规律:物体漂浮、轨迹违背重力、接触消失。标准去噪与流匹配目标仅拟合视觉分布,未显式惩罚此类物理违规。现有方法虽可改善物理一致性,但通常带来显著推理或训练开销。候选选择法需生成并评分多个视频,梯度引导需反复解码编码中间结果。生成器内部修正添加扰动与重去噪循环,而事后训练则需标注数据和额外优化。本文提出离流形精修(OMR),一种推理时方法,直接将世界模型反馈注入单条采样轨迹。在预定中段欧拉步骤中,通过适配器空间的V-JEPA 2.1意外能量梯度增强生成器速度。此外部修正可使潜在表示偏离原轨迹,向世界模型判定更合理的区域移动,随后从修正状态继续生成。小规模训练的潜向量到嵌入适配器确保梯度可计算,且生成器与世界模型均保持冻结。在固定400提示的VideoPhy-2详细子集上,OMR将联合语义一致与物理常识指标从47.0%提升至52.0%(+5.0个百分点,相对提升10.6%)。在另50提示的效率子集上,仅需1.71倍基线运行时间,远低于奖励/搜索类方法的乘法开销。
原文摘要 · Abstract (English)
Modern video generators routinely fail at physical dynamics: objects float, trajectories violate gravity, contacts vanish. Standard denoising and flow-matching objectives fit visual data distributions but do not explicitly penalize such physical violations. Existing remedies can improve physical consistency, but typically add substantial inference or training cost. Candidate-selection methods generate and score multiple videos, while gradient-based world-model guidance repeatedly decodes and re-encodes intermediate estimates. Generator-internal refinement adds perturbation and re-denoising loops, whereas post-training requires curated data and additional optimization. We propose Off-Manifold Refinement (OMR), an inference-time method that instead injects world-model feedback directly into a single sampling trajectory. During scheduled middle ODE steps, we augment the generator velocity with the gradient of an adapter-space V-JEPA 2.1 surprise energy. This external correction can move the latent away from the uncorrected sampling trajectory and toward regions ranked as more physically plausible by the frozen predictor, after which the generator continues rendering from the corrected state. A small trained latent-to-embedding adapter keeps the gradient tractable at inference, and both the video generator and the world model remain frozen. On our fixed 400-prompt VideoPhy-2 detailed subset, OMR lifts the joint Semantic-Adherence-and-Physical-Commonsense metric from 47.0% to 52.0% (+5.0pp absolute, +10.6% relative) over the base Wan2.2-T2V-A14B sampler. On a separate fixed 50-prompt efficiency subset, it requires $1.71 \times$ the base runtime rather than the multiplicative cost of reward/search alternatives. Project page: https://itruonghai.github.io/omr.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。