用电容视频实时精准追踪双手3D姿态，提升远程白板交互体验。
V-Hands: Touchscreen-based Hand Tracking for Remote Whiteboard Interaction
- 通过深度网络从电容帧中识别手部并定位关节。
- 结合约束逆运动学求解器，实现高精度3D手姿重建。
- 轻量设备部署，适合远程白板等协作场景。
在基于白板的远程通信中，绘图内容与手屏交互的无缝融合对沉浸式体验至关重要。现有方法或需笨重设备捕捉手势，或无法准确从电容图像中追踪手部姿态。本文提出一种实时方法，可从电容视频帧中精确追踪双手3D姿态。我们设计了一个深度神经网络，从电容帧中识别手部并推断关节位置，再通过约束逆运动学求解器恢复3D手姿。同时，我们搭建了高质量手屏交互数据采集装置，构建了更精确的同步电容视频与手姿数据集。所提方法在保持紧凑设备部署的同时，显著提升了电容帧下3D手部追踪的精度与稳定性。我们在白板远程交互场景中验证了方案的有效性，展示了优越性能。代码、模型与数据集已公开于 https://V-Hands.github.io。
原文摘要 · Abstract (English)
In whiteboard-based remote communication, the seamless integration of drawn content and hand-screen interactions is essential for an immersive user experience. Previous methods either require bulky device setups for capturing hand gestures or fail to accurately track the hand poses from capacitive images. In this paper, we present a real-time method for precise tracking 3D poses of both hands from capacitive video frames. To this end, we develop a deep neural network to identify hands and infer hand joint positions from capacitive frames, and then recover 3D hand poses from the hand-joint positions via a constrained inverse kinematic solver. Additionally, we design a device setup for capturing high-quality hand-screen interaction data and obtained a more accurate synchronized capacitive video and hand pose dataset. Our method improves the accuracy and stability of 3D hand tracking for capacitive frames while maintaining a compact device setup for remote communication. We validate our scheme design and its superior performance on 3D hand pose tracking and demonstrate the effectiveness of our method in whiteboard-based remote communication. Our code, model, and dataset are available at https://V-Hands.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。