用单目摄像头实现低成本机械臂的手势精准控制
Vision-Based Hand Shadowing for Robotic Manipulation via Inverse Kinematics
- 通过单目深度相机捕捉手部21个关键点，转换坐标后求解逆运动学生成机械臂指令
- 在结构化任务中成功率86.7%，比四种视觉语言动作模型表现更优
- 支持物理仿真预演，适合无标记手势控制研究与工程落地
低成本机械臂的遥操作仍因人类手势到机器人关节命令的映射困难而具有挑战性。本文提出一种基于单眼RGB-D相机（安装于3D打印眼镜）的离线手影逆运动学（IK）重定向流程。该流程使用MediaPipe Hands检测每只手的21个关键点，通过深度感知反投影至三维空间，转换至机器人坐标系后，求解阻尼最小二乘逆运动学问题，生成针对SO-ARM101机械臂（5个臂关节+1个夹爪）的关节指令。夹爪控制器通过拇指-食指几何关系映射抓取开度，并采用多级容错机制。动作在物理仿真中预览后再在真实机器人上执行。我们在结构化拾放基准测试中评估（5×5网格，每格10次抓取，3次独立运行），成功率达到86.7% ± 4.2%。并与四种基于领导者-追随者数据训练的视觉语言动作策略（ACT、SmolVLA、pi_0.5、GR00T N1.5）对比。提供定量误差分析：平均IK位置误差为36.4毫米，轨迹平滑性指标显示经EMA平滑后抖动降低57%-68%。还进行了平滑参数消融实验。在非结构化真实环境（超市、药店）测试中，因物体遮挡导致成功率降至9.3%。为缓解此问题，引入WiLoR作为替代手部检测器，相比MediaPipe将手部检测率提升8%，凸显无标记解析重定向的潜力与当前局限。
原文摘要 · Abstract (English)
Teleoperation of low-cost robotic manipulators remains challenging due to the difficulty of retargeting human hand motion to robot joint commands. We present an offline hand-shadowing inverse-kinematics (IK) retargeting pipeline driven by a single egocentric RGB-D camera mounted on 3D-printed glasses. The pipeline detects 21 hand landmarks per hand using MediaPipe Hands, deprojects them into 3D via depth sensing, transforms them into the robot coordinate frame, and solves a damped-least-squares IK problem to produce joint commands for the SO-ARM101 robot (5 arm + 1 gripper joints). A gripper controller maps thumb-index finger geometry to grasp aperture with a multi-level fallback hierarchy. Actions are previewed in a physics simulation before replay on the physical robot. We evaluate the pipeline on a structured pick-and-place benchmark (5-tile grid, 10 grasps per tile, 3 independent runs) achieving an 86.7% +/- 4.2% success rate, and compare it against four vision-language-action (VLA) policies (ACT, SmolVLA, pi_0.5, GR00T N1.5) trained on leader-follower teleoperation data. We provide a quantitative error analysis of the pipeline, reporting a mean IK position error of 36.4 mm, trajectory smoothness metrics showing 57-68% jerk reduction from EMA smoothing, and an ablation study over the smoothing parameter. We also test the pipeline in unstructured real-world environments (grocery store, pharmacy) and find that success is reduced to 9.3% due to hand occlusion by surrounding objects. To mitigate this, we integrate WiLoR as an alternative hand detector, achieving an 8% improvement in hand detection rate over MediaPipe, highlighting both the promise and current limitations of marker-free analytical retargeting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。