用几何一致性奖励提升视频生成的时空稳定性。
VIGOR: VIdeo Geometry-Oriented Reward for Temporal Generative Alignment
- 基于预训练几何模型,通过点对点重投影误差评估多视角一致性。
- 在无纹理和非语义区域过滤后,显著提升误差度量的鲁棒性。
- 支持推理时优化,无需重训即可增强开源视频模型质量。
视频扩散模型在训练中缺乏显式几何监督,导致生成视频出现物体形变、空间漂移和深度异常等不一致现象。为此,我们提出一种基于几何的奖励模型,利用预训练的几何基础模型,通过跨帧重投影误差评估多视角一致性。与以往在像素空间测量不一致的方法不同,我们的方法以逐点方式计算误差,获得更符合物理规律且抗噪能力强的度量。此外,我们设计了一种几何感知采样策略,过滤低纹理和非语义区域,聚焦于具有可靠对应关系的几何有意义区域,进一步提升鲁棒性。我们将该奖励模型应用于两个互补路径:通过SFT或强化学习对双向模型进行后训练;以及在推理时对因果视频模型(如流式视频生成器)进行测试时缩放优化,并以该奖励作为路径验证器。实验结果验证了设计的有效性,表明本方法相比其他变体具备更优的鲁棒性。通过实现高效的推理时缩放,该方法为提升开源视频模型提供了无需大量重训练资源的实用解决方案。
原文摘要 · Abstract (English)
Video diffusion models lack explicit geometric supervision during training, leading to inconsistency artifacts such as object deformation, spatial drift, and depth violations in generated videos. To address this limitation, we propose a geometry-based reward model that leverages pretrained geometric foundation models to evaluate multi-view consistency through cross-frame reprojection error. Unlike previous geometric metrics that measure inconsistency in pixel space, where pixel intensity may introduce additional noise, our approach conducts error computation in a pointwise fashion, yielding a more physically grounded and robust error metric. Furthermore, we introduce a geometry-aware sampling strategy that filters out low-texture and non-semantic regions, focusing evaluation on geometrically meaningful areas with reliable correspondences to improve robustness. We apply this reward model to align video diffusion models through two complementary pathways: post-training of a bidirectional model via SFT or Reinforcement Learning and inference-time optimization of a Causal Video Model (e.g., Streaming video generator) via test-time scaling with our reward as a path verifier. Experimental results validate the effectiveness of our design, demonstrating that our geometry-based reward provides superior robustness compared to other variants. By enabling efficient inference-time scaling, our method offers a practical solution for enhancing open-source video models without requiring extensive computational resources for retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。