用合成数据训练模型,实现快速准确的素描转3D人体姿态估计。
Sketch2PoseNet: Efficient and Generalized Sketch to 3D Human Pose Prediction
- 通过扩散模型生成12万对素描与3D姿态数据,解决标注稀缺问题。
- 在多个素描风格上达到更高精度,且推理速度远超传统方法。
- 适合动画师、影视制作人员快速将手绘草图转为3D角色动作。
从素描中进行3D人体姿态估计在计算机动画和影视制作中有广泛应用。由于素描具有抽象性和比例失真等特点,该任务面临独特挑战。以往方法受限于缺乏大规模素描-3D姿态标注,主要依赖启发式规则优化,耗时且泛化能力差。为此,本文提出一种“从合成中学习”的新策略:首先训练扩散模型,从3D人体姿态投影出的2D姿态生成素描图像,模拟素描中的比例失真,构建包含12万对标注的合成数据集SKEP-120K,涵盖多种素描风格。基于此数据集,设计端到端数据驱动框架,结合现有2D姿态检测器与生成式扩散先验提取素描特征,并使用前馈神经网络实现高效2D姿态估计。引入多种启发式损失函数,确保推导出的3D姿态与检测到的2D姿态几何一致,同时保持精确的自接触关系。定性、定量及主观评估均表明,本模型在估计精度与速度上显著优于此前方法。
原文摘要 · Abstract (English)
3D human pose estimation from sketches has broad applications in computer animation and film production. Unlike traditional human pose estimation, this task presents unique challenges due to the abstract and disproportionate nature of sketches. Previous sketch-to-pose methods, constrained by the lack of large-scale sketch-3D pose annotations, primarily relied on optimization with heuristic rules-an approach that is both time-consuming and limited in generalizability. To address these challenges, we propose a novel approach leveraging a "learn from synthesis" strategy. First, a diffusion model is trained to synthesize sketch images from 2D poses projected from 3D human poses, mimicking disproportionate human structures in sketches. This process enables the creation of a synthetic dataset, SKEP-120K, consisting of 120k accurate sketch-3D pose annotation pairs across various sketch styles. Building on this synthetic dataset, we introduce an end-to-end data-driven framework for estimating human poses and shapes from diverse sketch styles. Our framework combines existing 2D pose detectors and generative diffusion priors for sketch feature extraction with a feed-forward neural network for efficient 2D pose estimation. Multiple heuristic loss functions are incorporated to guarantee geometric coherence between the derived 3D poses and the detected 2D poses while preserving accurate self-contacts. Qualitative, quantitative, and subjective evaluations collectively show that our model substantially surpasses previous ones in both estimation accuracy and speed for sketch-to-pose tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。