用三摄像头低成本实现手指关节角度精准追踪，可实时控制机械手。
THETA: Triangulated Hand-State Estimation for Teleoperation and Automation in Robotic Hand Control
- 通过三视角三角测量与深度学习模型，从RGB图像推断手指关节角。
- 在48,000张图像上达到97.18%准确率，实现实时手部动作复制。
- 适合医疗、制造等需低成本远程操控的场景，部署简单易扩展。
机器人手的遥操作受限于深度相机和传感器手套的高成本，这些设备常用于估计手指相对关节位置（XYZ）。本文提出一种新型低成本方案，使用三个网络摄像头进行三角测量，以估算人类手指的相对关节角度（theta）。同时引入改良版DexHand（TheRobotStudio出品）展示THETA的实际应用。数据采集使用三个640x480分辨率摄像头以120度间隔布置，共生成超过48,000张RGB图像。关节角度通过手动测量指掌关节（MCP）、近端指间关节（PIP）和远端指间关节（DIP）中点确定。采用带有ResNet-50主干的DeepLabV3模型进行多尺度手部分割，随后经HSV滤波处理，输入由MobileNetV2构建的CNN分类器，该模型设计用于分层空间特征提取，并接收9通道输入张量以编码多视角手部表示。分类模型将分割后的手部视图映射为离散关节角，实现97.18%准确率、98.72%召回率、F1分数0.9274及0.8906精确率。实时推理中，THETA同步捕获帧、分割手部区域、滤波并生成9通道张量进行分类，预测结果通过串口发送至Arduino，驱动DexHand复现手部动作。未来工作将扩充数据集多样性，集成手腕追踪，并结合OpenAI-Vision等视觉技术。THETA有望为医疗、语言学和制造业提供低成本、易用的遥操作解决方案。
原文摘要 · Abstract (English)
The teleoperation of robotic hands is limited by the high costs of depth cameras and sensor gloves, commonly used to estimate hand relative joint positions (XYZ). We present a novel, cost-effective approach using three webcams for triangulation-based tracking to approximate relative joint angles (theta) of human fingers. We also introduce a modified DexHand, a low-cost robotic hand from TheRobotStudio, to demonstrate THETA's real-time application. Data collection involved 40 distinct hand gestures using three 640x480p webcams arranged at 120-degree intervals, generating over 48,000 RGB images. Joint angles were manually determined by measuring midpoints of the MCP, PIP, and DIP finger joints. Captured RGB frames were processed using a DeepLabV3 segmentation model with a ResNet-50 backbone for multi-scale hand segmentation. The segmented images were then HSV-filtered and fed into THETA's architecture, consisting of a MobileNetV2-based CNN classifier optimized for hierarchical spatial feature extraction and a 9-channel input tensor encoding multi-perspective hand representations. The classification model maps segmented hand views into discrete joint angles, achieving 97.18% accuracy, 98.72% recall, F1 Score of 0.9274, and a precision of 0.8906. In real-time inference, THETA captures simultaneous frames, segments hand regions, filters them, and compiles a 9-channel tensor for classification. Joint-angle predictions are relayed via serial to an Arduino, enabling the DexHand to replicate hand movements. Future research will increase dataset diversity, integrate wrist tracking, and apply computer vision techniques such as OpenAI-Vision. THETA potentially ensures cost-effective, user-friendly teleoperation for medical, linguistic, and manufacturing applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。