用多假设引导点云扩散,修复遮挡图像中的人体3D形状。
Multi-hypotheses Conditioned Point Cloud Diffusion for 3D Human Reconstruction from Occluded Images
- 基于多个假设的SMPL模型提取局部特征,融合后指导点云扩散重建。
- 在CAPE和MultiHuman数据集上,合成与真实遮挡下均优于现有方法。
- 适合需要高精度人体3D重建的虚拟试衣、动作捕捉等场景。
由于人-物或人-人交互导致严重遮挡,进行3D人体形状重建极具挑战。参数化模型(如SMPL(-X))基于人体形状统计,可表示完整人体,但仅限于裸露身体形态。基于隐函数的方法从参数化模型提取特征,利用人体先验知识,能捕捉衣物、头发等几何细节,但常难以处理对齐不良的SMPL网格,且单张RGB图像下难以修复遮挡区域。本文提出新框架MHCDIFF(多假设条件点云扩散),通过概率分布条件下的点云扩散模型,实现像素对齐的精细3D人体重建。其核心是提取多个假设的SMPL(-X)网格局部特征,并聚合为条件输入。实验表明,在CAPE和MultiHuman数据集上,该方法在合成与真实遮挡条件下,均超越基于SMPL、隐函数、点云扩散及其组合的多种最先进方法。代码已公开于https://donghwankim0101.github.io/projects/mhcdiff/。
原文摘要 · Abstract (English)
3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. Parametric models i.e., SMPL(-X), which are based on the statistics across human shapes, can represent whole human body shapes but are limited to minimally-clothed human shapes. Implicit-function-based methods extract features from the parametric models to employ prior knowledge of human bodies and can capture geometric details such as clothing and hair. However, they often struggle to handle misaligned parametric models and inpaint occluded regions given a single RGB image. In this work, we propose a novel pipeline, MHCDIFF, Multi-hypotheses Conditioned Point Cloud Diffusion, composed of point cloud diffusion conditioned on probabilistic distributions for pixel-aligned detailed 3D human reconstruction under occlusion. Compared to previous implicit-function-based methods, the point cloud diffusion model can capture the global consistent features to generate the occluded regions, and the denoising process corrects the misaligned SMPL meshes. The core of MHCDIFF is extracting local features from multiple hypothesized SMPL(-X) meshes and aggregating the set of features to condition the diffusion model. In the experiments on CAPE and MultiHuman datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions. Our code is publicly available at https://donghwankim0101.github.io/projects/mhcdiff/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。