arXiv:2409.02224cs.CVcs.HC2024-09CVPR被引 26

构建首个用于第一人称视觉的手部压力与姿态数据集,助力虚实交互研究。

EgoPressure: A Dataset for Hand Pressure and Pose Estimation in Egocentric Vision

  • 基于8相机多视角优化,重建高精度手部接触姿态
  • 提供每处接触点的压力强度标注,共5小时21人数据
  • 验证了压力与姿态联合建模对交互理解的互补价值

触觉接触与压力是理解人类操作物体的关键,对混合现实与机器人应用具有重要意义。然而,从第一人称视角估计这些交互仍面临挑战,主要因缺乏同时包含精准手部姿态和详细压力标注的综合性数据集。本文提出EgoPressure,一个新型第一人称数据集,捕捉精细的触碰与压力交互。该数据集为每个接触点提供高分辨率压力强度标注,并通过提出的多视角、序列化优化方法,从8相机采集系统中获得准确的手部姿态网格。数据包含21名参与者共5小时的交互记录,由1个头戴式和7个静止式Kinect摄像头同步采集,帧率为30 Hz,获取RGB图像与深度图。为支持未来研究与基准测试,我们提出了基于RGB图像估计外部表面施加压力的多个基线模型,含与不含手部姿态信息。进一步探索了手部网格与压力的联合估计。实验表明,压力与手部姿态在理解手物交互中具有互补性,推动了AR/VR与机器人研究中的手物交互建模。

原文摘要 · Abstract (English)

Touch contact and pressure are essential for understanding how humans interact with and manipulate objects, insights which can significantly benefit applications in mixed reality and robotics. However, estimating these interactions from an egocentric camera perspective is challenging, largely due to the lack of comprehensive datasets that provide both accurate hand poses on contacting surfaces and detailed annotations of pressure information. In this paper, we introduce EgoPressure, a novel egocentric dataset that captures detailed touch contact and pressure interactions. EgoPressure provides high-resolution pressure intensity annotations for each contact point and includes accurate hand pose meshes obtained through our proposed multi-view, sequence-based optimization method processing data from an 8-camera capture rig. Our dataset comprises 5 hours of recorded interactions from 21 participants captured simultaneously by one head-mounted and seven stationary Kinect cameras, which acquire RGB images and depth maps at 30 Hz. To support future research and benchmarking, we present several baseline models for estimating applied pressure on external surfaces from RGB images, with and without hand pose information. We further explore the joint estimation of the hand mesh and applied pressure. Our experiments demonstrate that pressure and hand pose are complementary for understanding hand-object interactions. ng of hand-object interactions in AR/VR and robotics research. Project page: \url{https://yiming-zhao.github.io/EgoPressure/}.

第一人称视觉手部压力姿态估计数据集

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。