用内部时空先验实现零样本刚性动作迁移,效果更稳定可控。
Motion Marionette: Rethinking Rigid Motion Transfer via Prior Guidance
- 通过统一3D空间构建共享的时空先验,不依赖外部几何或语义信息
- 生成的运动速度场支持可控视频生成,时间一致性显著提升
- 适用于多种物体,无需重新训练,适合高效视频创作场景
我们提出 Motion Marionette,一个零样本框架,将单目源视频中的刚性运动迁移至单视角目标图像。以往方法通常依赖几何、生成或仿真先验,但这些外部先验引入额外约束,导致泛化能力与时间一致性之间的权衡。为解决此问题,我们提出一种仅捕捉时空变换的内部先验,该先验在源视频与任意目标视频间共享。具体地,首先将源视频与目标图像统一映射至3D表示空间;从源视频中提取运动轨迹,构建独立于物体几何与语义的时空(SpaT)先验,编码随时间变化的相对空间关系。该先验进一步与目标物体结合,合成可控制的速度场,并通过基于位置的动力学(Position-Based Dynamics)进行优化,以减少伪影并增强视觉连贯性。最终生成的速度场可用于灵活高效的视频制作。实验表明,Motion Marionette 在多样化物体上具有强泛化能力,生成视频时间一致且与源运动高度匹配,并支持可控生成。
原文摘要 · Abstract (English)
We present Motion Marionette, a zero-shot framework for rigid motion transfer from monocular source videos to single-view target images. Previous works typically employ geometric, generative, or simulation priors to guide the transfer process, but these external priors introduce auxiliary constraints that lead to trade-offs between generalizability and temporal consistency. To address these limitations, we propose guiding the motion transfer process through an internal prior that exclusively captures the spatial-temporal transformations and is shared between the source video and any transferred target video. Specifically, we first lift both the source video and the target image into a unified 3D representation space. Motion trajectories are then extracted from the source video to construct a spatial-temporal (SpaT) prior that is independent of object geometry and semantics, encoding relative spatial variations over time. This prior is further integrated with the target object to synthesize a controllable velocity field, which is subsequently refined using Position-Based Dynamics to mitigate artifacts and enhance visual coherence. The resulting velocity field can be flexibly employed for efficient video production. Empirical results demonstrate that Motion Marionette generalizes across diverse objects, produces temporally consistent videos that align well with the source motion, and supports controllable video generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。