用事件相机提升第一人称手势识别,抗模糊光照,精度更高。
EgoEvGesture: Gesture Recognition Based on Egocentric Event Camera
- 设计轻量网络,用异步卷积和状态空间模块分离手部动作与头部晃动噪声。
- 在新数据集上达62.7%跨主体准确率,比现有方法高3.1%,且模型仅700万参数。
- 适合做低功耗、实时手势交互系统,尤其适用于动态环境下的可穿戴设备。
第一人称手势识别对自然人机交互至关重要,但传统基于RGB的方法在动态场景中易受运动模糊和光照变化影响。事件相机虽具备高动态范围与超低功耗优势,但现有基于帧的架构难以处理异步事件流。从第一人称视角看,事件数据同时包含头部运动和手部手势产生的事件,增加了识别难度。为此,我们提出EgoEvGesture,一种专为事件数据设计的新网络:(1)采用轻量级CNN结合非对称深度可分离卷积,减少参数量的同时保留时空特征;(2)引入即插即用的状态空间模型作为上下文模块,解耦头部运动噪声与手势动态;(3)设计无参的Bins-时序移位模块(BTSM),沿时间与分箱维度高效融合稀疏事件。我们还构建了首个大规模第一人称事件相机手势识别数据集EgoEvGesture。实验表明,该方法在未见主体测试下达到62.7%准确率,优于当前最优方案3.1%,仅需700万参数。在DVS128 Gesture数据集上更是达到97.0%准确率,验证了方法的有效性与泛化能力。代码与数据已开源。
原文摘要 · Abstract (English)
Egocentric gesture recognition is a pivotal technology for enhancing natural human-computer interaction, yet traditional RGB-based solutions suffer from motion blur and illumination variations in dynamic scenarios. While event cameras show distinct advantages in handling high dynamic range with ultra-low power consumption, existing RGB-based architectures face inherent limitations in processing asynchronous event streams due to their synchronous frame-based nature. Moreover, from an egocentric perspective, event cameras record data that includes events generated by both head movements and hand gestures, thereby increasing the complexity of gesture recognition. To address this, we propose a novel network architecture specifically designed for event data processing, incorporating (1) a lightweight CNN with asymmetric depthwise convolutions to reduce parameters while preserving spatiotemporal features, (2) a plug-and-play state-space model as context block that decouples head movement noise from gesture dynamics, and (3) a parameter-free Bins-Temporal Shift Module (BTSM) that shifts features along bins and temporal dimensions to fuse sparse events efficiently. We further establish the EgoEvGesture dataset, the first large-scale dataset for egocentric gesture recognition using event cameras. Experimental results demonstrate that our method achieves 62.7% accuracy tested on unseen subjects with only 7M parameters, 3.1% higher than state-of-the-art approaches. Notable misclassifications in freestyle motions stem from high inter-personal variability and unseen test patterns differing from training data. Moreover, our approach achieved a remarkable accuracy of 97.0% on the DVS128 Gesture, demonstrating the effectiveness and generalization capability of our method on public datasets. The dataset and models are made available at https://github.com/3190105222/EgoEv_Gesture.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。