用图像修复技术让角色艺术图动起来，还能精准控制镜头运动。
DreamDance: Animating Character Art via Inpainting Stable Gaussian Worlds
- 分两步修复：先生成带镜头运动的场景，再注入动态角色。
- 支持复杂镜头轨迹，角色动作稳定且与背景一致。
- 适合需要精细镜头控制的角色动画设计者。
本文提出DreamDance，一种基于图像修复的新角色艺术动画框架，可依据精确相机轨迹生成稳定、连贯的角色与场景运动。该方法将动画任务重构为两个基于修复的步骤：相机感知场景修复和姿态感知视频修复。第一步利用预训练图像修复模型，结合参考艺术图生成多视角场景图像，并优化一个大规模稳定高斯场，实现带有相机轨迹的粗略背景视频渲染。但渲染结果粗糙，仅体现场景运动。第二步训练了一个姿态感知视频修复模型，将动态角色注入场景视频并提升背景质量。该模型为基于DiT的视频生成模型，采用门控策略，自适应融合角色外观与姿态信息至基础背景视频。大量实验表明，DreamDance能生成高质量、一致的角色动画，并展现显著的相机动态效果。
原文摘要 · Abstract (English)
This paper presents DreamDance, a novel character art animation framework capable of producing stable, consistent character and scene motion conditioned on precise camera trajectories. To achieve this, we re-formulate the animation task as two inpainting-based steps: Camera-aware Scene Inpainting and Pose-aware Video Inpainting. The first step leverages a pre-trained image inpainting model to generate multi-view scene images from the reference art and optimizes a stable large-scale Gaussian field, which enables coarse background video rendering with camera trajectories. However, the rendered video is rough and only conveys scene motion. To resolve this, the second step trains a pose-aware video inpainting model that injects the dynamic character into the scene video while enhancing background quality. Specifically, this model is a DiT-based video generation model with a gating strategy that adaptively integrates the character's appearance and pose information into the base background video. Through extensive experiments, we demonstrate the effectiveness and generalizability of DreamDance, producing high-quality and consistent character animations with remarkable camera dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。