arXiv:2409.11104cs.CV2024-09ECCV被引 3

用RGB图生成深度信息,提升3D人体姿态估计精度

Depth-based Privileged Information for Boosting 3D Human Pose Estimation on RGB

  • 训练时用深度数据指导,推理时仅需RGB图像
  • 在有限数据下仍显著提升3D姿态估计准确率
  • 适合缺乏深度设备的实拍场景应用

尽管计算机视觉研究取得进展,从单张RGB图像估计3D人体姿态仍是挑战,因为多个3D姿态可能对应相同的2D投影。深度数据可通过提供物体与相机间距离的额外约束来消除歧义。然而,精确深度数据通常局限于室内场景,且依赖特定设备,限制了泛化能力。本文提出一种方法,能在不牺牲广泛适用性的情况下利用深度信息的优势。该方法基于热力图的3D姿态估计器,采用特权信息范式,在推理时仅凭RGB帧即可‘幻觉’出深度信息。具体而言,深度信息仅在训练阶段使用,通过强制基于RGB的幻觉网络学习与仅在深度数据上预训练的主干网络相似特征。实验表明,该范式显著提升模型性能,仅用RGB图像即可高效提取深度信息。

原文摘要 · Abstract (English)

Despite the recent advances in computer vision research, estimating the 3D human pose from single RGB images remains a challenging task, as multiple 3D poses can correspond to the same 2D projection on the image. In this context, depth data could help to disambiguate the 2D information by providing additional constraints about the distance between objects in the scene and the camera. Unfortunately, the acquisition of accurate depth data is limited to indoor spaces and usually is tied to specific depth technologies and devices, thus limiting generalization capabilities. In this paper, we propose a method able to leverage the benefits of depth information without compromising its broader applicability and adaptability in a predominantly RGB-camera-centric landscape. Our approach consists of a heatmap-based 3D pose estimator that, leveraging the paradigm of Privileged Information, is able to hallucinate depth information from the RGB frames given at inference time. More precisely, depth information is used exclusively during training by enforcing our RGB-based hallucination network to learn similar features to a backbone pre-trained only on depth data. This approach proves to be effective even when dealing with limited and small datasets. Experimental results reveal that the paradigm of Privileged Information significantly enhances the model's performance, enabling efficient extraction of depth information by using only RGB images.

3D姿态估计深度幻觉特权信息RGB-only

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。