从单目视频中恢复动物3D姿态,无需真实3D标注
L3D-Pose: Lifting Pose for 3D Avatars from a Single Camera in the Wild
- 用合成数据+骨骼动画生成3D姿态标签,解决真实数据难获取问题
- 设计注意力MLP将2D姿态转为3D,可适配野外复杂场景
- 提出查找表方法提升任意角色的动作重定向精度,适合动物动画应用
尽管2D姿态估计已能解析动物和灵长类的运动,但因缺乏深度信息而受限。3D姿态估计通过引入空间深度提供更全面解决方案,然而在自然环境中为动物构建大规模3D姿态数据集极具挑战性。为此,我们提出一种混合方法:利用带骨骼的虚拟角色与合成流程生成所需3D标注数据。所提方法采用简单的基于注意力的MLP网络,将2D姿态转换为3D,且不依赖输入图像,确保在自然环境中的可扩展性。此外,我们发现现有解剖关键点检测器在将姿态重定向至任意角色时表现不足。为此,我们基于合成多样化动作数据,提出一种基于查找表的深度姿态估计方法。实验表明该查找表方法在动作重定向上高效准确。整体上,我们构建了一个系统化的合成数据框架,实现从2D到3D姿态提升,并将其应用于野外场景动作向任意角色的迁移。
原文摘要 · Abstract (English)
While 2D pose estimation has advanced our ability to interpret body movements in animals and primates, it is limited by the lack of depth information, constraining its application range. 3D pose estimation provides a more comprehensive solution by incorporating spatial depth, yet creating extensive 3D pose datasets for animals is challenging due to their dynamic and unpredictable behaviours in natural settings. To address this, we propose a hybrid approach that utilizes rigged avatars and the pipeline to generate synthetic datasets to acquire the necessary 3D annotations for training. Our method introduces a simple attention-based MLP network for converting 2D poses to 3D, designed to be independent of the input image to ensure scalability for poses in natural environments. Additionally, we identify that existing anatomical keypoint detectors are insufficient for accurate pose retargeting onto arbitrary avatars. To overcome this, we present a lookup table based on a deep pose estimation method using a synthetic collection of diverse actions rigged avatars perform. Our experiments demonstrate the effectiveness and efficiency of this lookup table-based retargeting approach. Overall, we propose a comprehensive framework with systematically synthesized datasets for lifting poses from 2D to 3D and then utilize this to re-target motion from wild settings onto arbitrary avatars.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。