arXiv:2512.11792cs.CV2025-12被引 1

用视频追踪模型提炼运动结构先验,提升视频生成真实感

Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation

论文配图:Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation
图 1 · 摘自论文原文
  • 从自回归追踪模型中提取结构保持的运动先验,注入扩散模型
  • 在VBench上得分95.51,比基线高2.60%,FVD降低超21%
  • 适合需要高物理合理性视频生成的研究与应用

现实是刚性约束与可变形结构的动态平衡。对于视频生成模型而言,需生成既保真又保持结构一致的运动。尽管扩散模型已有进展,但对人类、动物等关节与可变形物体,仍难以生成合理的结构保持运动,单纯扩大训练数据无法解决物理不合理过渡问题。现有方法依赖外部不完美模型提取的噪声运动表示(如光流或骨骼)。为此,本文提出一种算法,将自回归视频追踪模型(SAM2)中的结构保持运动先验蒸馏至双向扩散模型(CogVideoX),构建SAM2VideoX。该模型包含两项创新:(1)双向特征融合模块,从类似SAM2的递归模型中提取全局结构保持运动先验;(2)局部格拉姆流损失,对齐局部特征协同运动方式。在VBench和人类评估中,SAM2VideoX相较基线显著提升:VBench得分95.51,优于REPA(92.91)2.60%;FVD降至360.57,分别较REPA和LoRA微调降低21.20%与22.46%;人类偏好达71.4%。项目主页见https://sam2videox.github.io/

原文摘要 · Abstract (English)

Reality is a dance between rigid constraints and deformable structures. For video models, that means generating motion that preserves fidelity as well as structure. Despite progress in diffusion models, producing realistic structure-preserving motion remains challenging, especially for articulated and deformable objects such as humans and animals. Scaling training data alone, so far, has failed to resolve physically implausible transitions. Existing approaches rely on conditioning with noisy motion representations, such as optical flow or skeletons extracted using an external imperfect model. To address these challenges, we introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX). With our method, we train SAM2VideoX, which contains two innovations: (1) a bidirectional feature fusion module that extracts global structure-preserving motion priors from a recurrent model like SAM2; (2) a Local Gram Flow loss that aligns how local features move together. Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains (+2.60\% on VBench, 21-22\% lower FVD, and 71.4\% human preference) over prior baselines. Specifically, on VBench, we achieve 95.51\%, surpassing REPA (92.91\%) by 2.60\%, and reduce FVD to 360.57, a 21.20\% and 22.46\% improvement over REPA- and LoRA-finetuning, respectively. The project website can be found at https://sam2videox.github.io/ .

视频生成结构保持扩散模型运动先验

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。