arXiv:2605.12297cs.CVcs.RO2026-05

用双目事件相机实现低光和遮挡下双手姿态与手势的精准识别。

EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras

论文配图:EgoEV-HandPose: Egocentric 3D Hand Pose Estimation and Gesture Recognition with Stereo Event Cameras
图 1 · 摘自论文原文
  • 通过鸟瞰视角融合与迭代重投影优化,解决事件流中的深度模糊问题。
  • 在5419段真实场景数据上达到30.54mm MPJPE和86.87%手势识别率。
  • 首个大规模双目事件相机手部数据集,适合虚实交互与机器人应用。

主观视角下的3D手部姿态估计与手势识别对沉浸式增强/虚拟现实、人机交互及机器人至关重要。然而,传统帧式相机存在运动模糊和动态范围有限的问题,现有基于事件的方法受限于自我运动干扰、单目深度模糊以及缺乏大规模真实世界双目数据集。为此,我们提出EgoEV-HandPose,一种端到端框架,可从双目事件流中联合进行3D双手姿态估计与手势识别。核心是KeypointBEV模块,将特征提升至标准鸟瞰视图空间,并通过迭代重投影引导的精炼循环逐步消除深度不确定性并强制满足运动学一致性。此外,我们构建了EgoEVHands,首个大规模真实世界双目事件相机手部感知数据集,包含5,419段标注序列,覆盖38类手势,在不同光照条件下提供密集的3D/2D关键点标注。大量实验表明,EgoEV-HandPose在低光和双手遮挡场景中显著优于基于RGB的双目方法和已有事件相机方法,达到30.54mm的MPJPE和86.87%的Top-1手势识别准确率,树立了事件基主观感知的新基准。数据集与源代码将公开发布于https://github.com/ZJUWang01/EgoEV-HandPose。

原文摘要 · Abstract (English)

Egocentric 3D hand pose estimation and gesture recognition are essential for immersive augmented/virtual reality, human-computer interaction, and robotics. However, conventional frame-based cameras suffer from motion blur and limited dynamic range, while existing event-based methods are hindered by ego-motion interference, monocular depth ambiguity, and the lack of large-scale real-world stereo datasets. To overcome these limitations, we propose EgoEV-HandPose, an end-to-end framework for joint 3D bimanual pose estimation and gesture recognition from stereo event streams. Central to our approach is KeypointBEV, a flexible stereo fusion module that lifts features into a canonical bird's-eye-view space and employs an iterative reprojection-guided refinement loop to progressively resolve depth uncertainty and enforce kinematic consistency. In addition, we introduce EgoEVHands, the first large-scale real-world stereo event-camera dataset for egocentric hand perception, containing 5,419 annotated sequences with dense 3D/2D keypoints across 38 gesture classes under varying illumination. Extensive experiments demonstrate that EgoEV-HandPose achieves state-of-the-art performance with an MPJPE of 30.54mm and 86.87% Top-1 gesture recognition accuracy, significantly outperforming RGB-based stereo and prior event-camera methods, particularly in low-light and bimanual occlusion scenarios, thereby setting a new benchmark for event-based egocentric perception. The established dataset and source code will be publicly released at https://github.com/ZJUWang01/EgoEV-HandPose.

事件相机手势识别3D姿态双目感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。