利用深度数据提升单视角下人体网格重建精度，解决数据稀缺难题。
3D Human Mesh Estimation from Single View RGBD
- 用动作捕捉数据生成虚拟单视角RGBD图像，解决真实标注数据少的问题。
- 通过掩码自编码器恢复被遮挡部位，实现完整人体网格重建，误差低至16.8mm。
- 适合做基于深度相机的人体建模研究，尤其在遮挡严重场景表现优异。
尽管单视角RGB图像的3D人体网格估计已取得显著进展，但能提供额外深度信息的RGBD相机仍被低估。本文提出一种从单视角RGBD图像精确重建3D人体网格的方法，充分利用低成本且普及的RGBD相机。现有全监督方法需同时具备RGBD图像与3D网格标签的数据集，但此类数据收集成本高、规模小、姿态与体型多样性不足。为此，我们借助现有的动作捕捉（MoCap）数据集，通过投影生成虚拟单视角下的部分深度图，模拟真实传感器输出。在此基础上训练一个掩码自编码器，以完成缺失的网格结构。推理时，将传感器获取的深度值与模板人体网格顶点对齐，形成部分可见网格，再通过模型恢复被遮挡区域，最终获得完整人体网格。该方法命名为M³（Masked Mesh Modeling），在SURREAL和CAPE数据集上分别达到16.8 mm和22.0 mm的每顶点误差（PVE），优于使用完整点云输入的现有方法；在BEHAVE数据集上达到70.9 PVE，较最新基于RGB的方法提升18.4 mm，验证了深度信息的有效性。代码将公开。
原文摘要 · Abstract (English)
Despite significant progress in 3D human mesh estimation from RGB images; RGBD cameras, offering additional depth data, remain underutilized. In this paper, we present a method for accurate 3D human mesh estimation from a single RGBD view, leveraging the affordability and widespread adoption of RGBD cameras for real-world applications. A fully supervised approach for this problem, requires a dataset with RGBD image and 3D mesh label pairs. However, collecting such a dataset is costly and challenging, hence, existing datasets are small, and limited in pose and shape diversity. To overcome this data scarcity, we leverage existing Motion Capture (MoCap) datasets. We first obtain complete 3D meshes from the body models found in MoCap datasets, and create partial, single-view versions of them by projection to a virtual camera. This simulates the depth data provided by an RGBD camera from a single viewpoint. Then, we train a masked autoencoder to complete the partial, single-view mesh. During inference, our method, which we name as M$^3$ for ``Masked Mesh Modeling'', matches the depth values coming from the sensor to vertices of a template human mesh, which creates a partial, single-view mesh. We effectively recover parts of the 3D human body mesh model that are not visible, resulting in a full body mesh. M$^3$ achieves 16.8 mm and 22.0 mm per-vertex-error (PVE) on the SURREAL and CAPE datasets, respectively; outperforming existing methods that use full-body point clouds as input. We obtain a competitive 70.9 PVE on the BEHAVE dataset, outperforming a recently published RGB based method by 18.4 mm, highlighting the usefulness of depth data. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。