用野外视频生成逼真动物动作,支持文本图像双重控制。
Kirin: Animal Motion Generation from In-the-Wild Video

- 从野外视频重建3D动作,构建跨物种动作数据集
- 基于视频-文本-动作三元组训练生成模型,实现多物种动作合成
- 自动绑定并渲染3D模型,适合动画与行为研究应用
理解动物运动对建模动物行为与生物力学至关重要,但因高质量动作数据稀缺,该领域进展远落后于人类动作研究。人类动作可在受控环境采集,而大多数动物难以实现,导致数据集规模小、领域局限,限制了动画等下游应用。为此,我们提出Kirin框架,可从视频中重建动作,大规模学习动作先验,并直接生成可用于动画资产的逼真动作。利用大量野外动物视频,我们重建3D动作序列,并配以描述性文字,构建了首个面向四足动物的大规模对齐视频-文本-动作数据集AiM3D。基于此数据集,我们开发了一种视觉引导的动作生成模型,同时以文本和图像为条件,生成多样化的跨物种真实动作。最后,通过调用现成的图像到3D模型工具,自动为3D网格绑定并驱动动画,生成可直接渲染的动画动物。整体框架与数据集为大规模、文本与图像条件下的动物动作生成与动画奠定了新基础。
原文摘要 · Abstract (English)
Understanding animal motion is fundamental to modeling animal behavior and biomechanics, yet progress in this area lags far behind human motion research due to the scarcity of high-quality motion data. While human motion can be captured in controlled environments, it is impractical for most animal species, resulting in small, domain-limited datasets that restrict downstream applications such as animation. To address this challenge, we introduce Kirin, a framework that reconstructs motion from video, learns motion priors at scale, and generates realistic motion that can be directly applied to animated assets. Using large collections of in-the-wild animal videos, we reconstruct 3D motion sequences and pair them with captions to create AiM3D, the first large-scale dataset offering aligned video-text-motion tuples for quadruped animals. Building on this dataset, we develop a visual-guided motion generation model that conditions on both text and image to guide the generation of realistic motion across diverse animal species. Finally, by leveraging an off-the-shelf image-to-3D model, we automatically rig and animate 3D meshes using generated motion, producing ready-to-render animated animals. Together, our dataset and framework establish a new foundation for large-scale, text and image conditioned animal motion generation and animation. Project page: https://kirin-ani.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。