用3D感知实现相机与物体协同精细控制的图像动画生成
Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representation
- 构建3D感知运动表示，实现相机与物体协同控制
- 支持多种运动相关视频生成任务，统一灵活
- 用户指令可直观转化为视觉变化，适合交互式创作
运动可控的图像动画是具有广泛应用潜力的基础任务。现有方法在通过不同运动表示控制相机或物体运动方面取得进展，但仍难以支持相机与物体运动的协同控制及自适应控制粒度。为此，本文提出3D感知运动表示，并构建名为Perception-as-Control的图像动画框架，实现细粒度协同运动控制。具体而言，从参考图像构建3D感知运动表示，根据用户指令进行操作，并从不同视角感知其变化。如此，相机与物体运动被转化为直观且一致的视觉变化。随后，该框架将感知结果作为运动控制信号，以统一且灵活的方式支持多种运动相关的视频合成任务。实验表明该方法具有优越性。
原文摘要 · Abstract (English)
Motion-controllable image animation is a fundamental task with a wide range of potential applications. Recent works have made progress in controlling camera or object motion via various motion representations, while they still struggle to support collaborative camera and object motion control with adaptive control granularity. To this end, we introduce 3D-aware motion representation and propose an image animation framework, called Perception-as-Control, to achieve fine-grained collaborative motion control. Specifically, we construct 3D-aware motion representation from a reference image, manipulate it based on interpreted user instructions, and perceive it from different viewpoints. In this way, camera and object motions are transformed into intuitive and consistent visual changes. Then, our framework leverages the perception results as motion control signals, enabling it to support various motion-related video synthesis tasks in a unified and flexible way. Experiments demonstrate the superiority of the proposed approach. For more details and qualitative results, please refer to our anonymous project webpage: https://chen-yingjie.github.io/projects/Perception-as-Control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。