arXiv:2509.13883cs.CV2025-09

用事件相机实现低延迟、低功耗的首人称视角手部三维追踪。

EvHand-FPV: Efficient Event-Based 3D Hand Tracking from First-Person View

  • 基于事件相机与几何线索定位手部区域,减少计算量。
  • 参数量降低89%,推理算力减少89%,2D追踪准确率提升至0.85。
  • 适合资源受限的扩展现实设备实时手部追踪应用。

手部追踪在直观交互中潜力巨大,但传统帧基方法难以兼顾精度、低延迟与能效,尤其在资源受限的扩展现实(XR)设备中。事件相机以微秒级时间分辨率和毫瓦级功耗异步感知亮度变化。本文提出 EvHand-FPV,一个轻量级框架,用于单事件相机的首人称视角3D手部追踪。构建了一个事件数据集,结合合成训练数据(带3D标签)与真实事件数据(带2D标签)以弥补首人称基准的稀缺。该方法引入基于腕部的感兴趣区域(ROI),通过几何线索定位手部,结合端到端嵌入ROI偏移的映射策略,避免显式重建以降低计算开销,并采用多任务学习策略,在不增加测试开销的前提下提升表征能力。在真实FPV测试集上,2D-AUCp从0.77提升至0.85,参数量由11.2M降至1.2M(减少89%),每推理前向计算量从1.648G降至0.185G(减少89%)。在合成数据上保持0.84的3D-AUCp。结果表明该方法在准确性和效率上均适合设备端的XR应用。数据集与代码见 https://github.com/zen5x5/EvHand-FPV。

原文摘要 · Abstract (English)

Hand tracking holds great promise for intuitive interaction paradigms, but frame-based methods often struggle to meet the requirements of accuracy, low latency, and energy efficiency, especially in resource-constrained settings such as Extended Reality (XR) devices. Event cameras provide $μ$s-level temporal resolution at mW-level power by asynchronously sensing brightness changes. In this work, we present EvHand-FPV, a lightweight framework for egocentric First-Person-View 3D hand tracking from a single event camera. We construct an event-based FPV dataset that couples synthetic training data with 3D labels and real event data with 2D labels for evaluation to address the scarcity of egocentric benchmarks. EvHand-FPV also introduces a wrist-based region of interest (ROI) that localizes the hand region via geometric cues, combined with an end-to-end mapping strategy that embeds ROI offsets into the network to reduce computation without explicit reconstruction, and a multi-task learning strategy with an auxiliary geometric feature head that improves representations without test-time overhead. On our real FPV test set, EvHand-FPV improves 2D-AUCp from 0.77 to 0.85 while reducing parameters from 11.2M to 1.2M by 89% and FLOPs per inference from 1.648G to 0.185G by 89%. It also maintains a competitive 3D-AUCp of 0.84 on synthetic data. These results demonstrate accurate and efficient egocentric event-based hand tracking suitable for on-device XR applications. The dataset and code are available at https://github.com/zen5x5/EvHand-FPV.

手部追踪事件相机轻量化XR

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。