arXiv:2503.05995cs.RO2025-03

实时高精度手部姿态与网格重建,速度达72帧/秒

ReJSHand: Efficient Real-Time Hand Pose Estimation and Mesh Reconstruction Using Refined Joint and Skeleton Features

  • 通过优化关节与骨骼特征,融合自注意力与坐标注意力机制
  • 在FreiHand数据集上实现6.3毫米的3D关键点误差,72帧/秒速度
  • 适合需要快速响应的机器人操作与人机交互场景

准确的手部姿态估计对机器人和人机交互中的灵巧操作至关重要。本文提出ReJSHand(精炼关节与骨骼特征),一种面向实时手部姿态估计与网格重建的先进网络框架。该模型通过2D关键点生成器、3D关键点生成器、扩展模块和特征交互模块,从2D图像精确重构3D手部姿态。引入多头自注意力机制与坐标注意力层,提升特征表达能力,通过复杂特征映射与线性变换高效生成手部网格顶点。在FreiHand数据集上的评估显示,ReJSHand实现72帧/秒的推理速度,同时保持6.3毫米的PA-MPJPE和6.4毫米的PA-MPVPE误差。此外,其F@05达到0.756,F@15达0.984,优于现有方法,确立了在机器人手部估计领域的领先地位。源代码已开源。

原文摘要 · Abstract (English)

Accurate hand pose estimation is vital in robotics, advancing dexterous manipulation in human-computer interaction. Toward this goal, this paper presents ReJSHand (which stands for Refined Joint and Skeleton Features), a cutting-edge network formulated for real-time hand pose estimation and mesh reconstruction. The proposed framework is designed to accurately predict 3D hand gestures under real-time constraints, which is essential for systems that demand agile and responsive hand motion tracking. The network's design prioritizes computational efficiency without compromising accuracy, a prerequisite for instantaneous robotic interactions. Specifically, ReJSHand comprises a 2D keypoint generator, a 3D keypoint generator, an expansion block, and a feature interaction block for meticulously reconstructing 3D hand poses from 2D imagery. In addition, the multi-head self-attention mechanism and a coordinate attention layer enhance feature representation, streamlining the creation of hand mesh vertices through sophisticated feature mapping and linear transformation. Regarding performance, comprehensive evaluations on the FreiHand dataset demonstrate ReJSHand's computational prowess. It achieves a frame rate of 72 frames per second while maintaining a PA-MPJPE (Position-Accurate Mean Per Joint Position Error) of 6.3 mm and a PA-MPVPE (Position-Accurate Mean Per Vertex Position Error) of 6.4 mm. Moreover, our model reaches scores of 0.756 for F@05 and 0.984 for F@15, surpassing modern pipelines and solidifying its position at the forefront of robotic hand pose estimators. To facilitate future studies, we provide our source code at ~\url{https://github.com/daishipeng/ReJSHand}.

手部姿态实时估计网格重建机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。