提出AirGlove,让视觉模型快速适应新传感手套的3D手部追踪。
AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Gloves
- 利用已有手套数据,迁移学习泛化新手套的外观特征。
- 在多款传感手套上实现显著性能提升,零样本和微调均有效。
- 适合做机器人遥操作与触觉反馈系统的研究者参考。
传感手套在远程操作和机器人策略学习中日益重要,能提供速度、加速度和触觉反馈等丰富信号。传统传感器驱动的手部追踪依赖角速度、重力方向等信号,但受传感器质量和校准影响较大。近年来基于视觉的方法在裸手追踪上表现优异,但对具有显著外观差异的传感手套仍缺乏研究。本文首次系统评估了视觉方法在传感手套上的表现,发现现有裸手模型因外观差异导致性能大幅下降。为此,我们提出AirGlove,利用已有的手套数据,将学习到的手套表征泛化至新设计的手套,仅需少量数据即可实现高效适配。实验表明,AirGlove在多种传感手套上均显著优于对比方法,有效提升了手部姿态估计性能。
原文摘要 · Abstract (English)
Sensing gloves have become important tools for teleoperation and robotic policy learning as they are able to provide rich signals like speed, acceleration and tactile feedback. A common approach to track gloved hands is to directly use the sensor signals (e.g., angular velocity, gravity orientation) to estimate 3D hand poses. However, sensor-based tracking can be restrictive in practice as the accuracy is often impacted by sensor signal and calibration quality. Recent advances in vision-based approaches have achieved strong performance on human hands via large-scale pre-training, but their performance on gloved hands with distinct visual appearances remains underexplored. In this work, we present the first systematic evaluation of vision-based hand tracking models on gloved hands under both zero-shot and fine-tuning setups. Our analysis shows that existing bare-hand models suffer from substantial performance degradation on sensing gloves due to large appearance gap between bare-hand and glove designs. We therefore propose AirGlove, which leverages existing gloves to generalize the learned glove representations towards new gloves with limited data. Experiments with multiple sensing gloves show that AirGlove effectively generalizes the hand pose models to new glove designs and achieves a significant performance boost over the compared schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。