arXiv:2509.16949cs.CV2025-09被引 2

用真实RGB图像预训练事件相机手部姿态估计,提升动态手部捕捉效果。

Leveraging RGB Images for Pre-Training of Event-Based Hand Pose Estimation

  • 通过分解手部运动为小步动作生成更真实的伪事件数据
  • 在EvRealHands数据集上实现最高24%的性能提升
  • 仅需少量标注数据即可良好微调,适合实际应用

本文提出RPEP,首个利用标注RGB图像与未配对无标签事件数据进行预训练的事件相机3D手部姿态估计方法。事件数据具有高时间分辨率和低延迟优势,但手部姿态估计受限于标注数据稀缺。为此,我们复用真实RGB数据集,构建伪事件-图像对,将事件数据与RGB图像的真实姿态对齐。现有伪事件生成方法假设物体静止,难以处理动态手部运动。RPEP引入新生成策略,将手部运动分解为逐步小动作,以捕捉关节运动的时序变化,生成更逼真的事件数据。此外,引入运动反转约束,通过反向运动正则化事件生成过程。大量实验表明,预训练模型在真实事件数据上显著优于当前最佳方法,在EvRealHands数据集上最高提升达24%。同时,仅需极少标注样本即可实现良好微调,适用于实际部署。

原文摘要 · Abstract (English)

This paper presents RPEP, the first pre-training method for event-based 3D hand pose estimation using labeled RGB images and unpaired, unlabeled event data. Event data offer significant benefits such as high temporal resolution and low latency, but their application to hand pose estimation is still limited by the scarcity of labeled training data. To address this, we repurpose real RGB datasets to train event-based estimators. This is done by constructing pseudo-event-RGB pairs, where event data is generated and aligned with the ground-truth poses of RGB images. Unfortunately, existing pseudo-event generation techniques assume stationary objects, thus struggling to handle non-stationary, dynamically moving hands. To overcome this, RPEP introduces a novel generation strategy that decomposes hand movements into smaller, step-by-step motions. This decomposition allows our method to capture temporal changes in articulation, constructing more realistic event data for a moving hand. Additionally, RPEP imposes a motion reversal constraint, regularizing event generation using reversed motion. Extensive experiments show that our pre-trained model significantly outperforms state-of-the-art methods on real event data, achieving up to 24% improvement on EvRealHands. Moreover, it delivers strong performance with minimal labeled samples for fine-tuning, making it well-suited for practical deployment.

事件相机手部姿态预训练伪数据

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。