将手绘分镜直接转为3D动画,支持精准动作控制。
Sketch2Anim: Towards Transferring Sketch Storyboards into 3D Animation
- 用3D关键帧和动作词联合控制生成动作
- 实现手绘分镜到3D动画的高质量转换
- 适合动画师快速原型设计与编辑
分镜是制作3D动画的常用手段。动画师通常以2D分镜为参考,通过反复试错构建理想3D动画,该过程依赖高超技能,且耗时费力。因此,亟需自动化方法将2D分镜直接转化为3D动画。该任务迄今研究较少。受运动扩散模型进展启发,本文从条件运动生成角度提出Sketch2Anim,包含两个核心模块:草图理解与运动生成。由于2D草图与3D动作之间存在显著领域差距,不直接以2D输入为条件,而是设计一个3D条件运动生成器,同时利用3D关键帧、关节轨迹和动作词,实现精确细粒度的动作控制。此外,提出一种神经映射器,在共享嵌入空间中对齐用户提供的2D草图与其对应的3D关键帧和轨迹,首次实现2D草图对运动生成的直接控制。所提方法成功将分镜转换为高质量3D动作,并具备天然的3D动画编辑能力,得益于多条件生成器的灵活性。大量实验与用户感知评估验证了方法的有效性。
原文摘要 · Abstract (English)
Storyboarding is widely used for creating 3D animations. Animators use the 2D sketches in storyboards as references to craft the desired 3D animations through a trial-and-error process. The traditional approach requires exceptional expertise and is both labor-intensive and time-consuming. Consequently, there is a high demand for automated methods that can directly translate 2D storyboard sketches into 3D animations. This task is under-explored to date and inspired by the significant advancements of motion diffusion models, we propose to address it from the perspective of conditional motion synthesis. We thus present Sketch2Anim, composed of two key modules for sketch constraint understanding and motion generation. Specifically, due to the large domain gap between the 2D sketch and 3D motion, instead of directly conditioning on 2D inputs, we design a 3D conditional motion generator that simultaneously leverages 3D keyposes, joint trajectories, and action words, to achieve precise and fine-grained motion control. Then, we invent a neural mapper dedicated to aligning user-provided 2D sketches with their corresponding 3D keyposes and trajectories in a shared embedding space, enabling, for the first time, direct 2D control of motion generation. Our approach successfully transfers storyboards into high-quality 3D motions and inherently supports direct 3D animation editing, thanks to the flexibility of our multi-conditional motion generator. Comprehensive experiments and evaluations, and a user perceptual study demonstrate the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。