用单目视频实现精准手指生物力学追踪，助力临床康复评估
Monocular Biomechanical Tracking of Fingers with Inverse Kinematics to Foundation Models
- 结合SAM 3D Body与逆运动学优化，构建全身生物力学模型
- 手指关节角度误差约10度，手部位置误差约6毫米
- 支持真实视频场景，适合医疗康复与动作分析应用
从视频中精确追踪手部与手指运动在临床中对日常生活活动监测和关节活动范围测量具有重要意义，但单目视频获取手部生物力学的方法仍不成熟。本文提出一种方法，将SAM 3D Body基础模型与逆运动学优化结合，集成于全身体生物力学模型中，从单视角视频中提取符合解剖约束的手指关节角度。我们把SAM 3D Body从PyTorch移植到JAX，以实现与MuJoCo-MJX的集成，支持GPU加速优化，并开发了新的Momentum Human Rig（MHR）输出与生物力学模型标记间的映射关系。在7名参与者完成多种手部姿势和物体操作任务的4,590帧多相机三维重建数据上验证，经普鲁克斯特对齐后，手指关节角度误差约为10度，手部位置误差约为6毫米。结果在不同摄像头视角下保持一致，且对多视角视频生成参考值的方法不敏感。本工作将单目生物力学分析拓展至精细手指追踪，使从常见视频中获取手部运动定量特征成为可能。
原文摘要 · Abstract (English)
Accurate hand and finger tracking from video has significant clinical applications for monitoring activities of daily living and measuring range of motion, yet monocular video approaches for obtaining hand biomechanics remain under-developed. We present a method that combines the SAM 3D Body foundation model with inverse kinematics optimization in a full-body biomechanical model to extract anatomically-constrained finger joint angles from single-view video. We port SAM 3D Body from PyTorch to JAX for integration with MuJoCo-MJX, enabling GPU-accelerated optimization, and develop a novel mapping between the Momentum Human Rig (MHR) outputs and biomechanical model markers. Validation against 8-camera multiview reconstruction on 4,590 frames from 7 participants performing a variety of hand poses and object manipulation tasks shows finger joint angle errors of approximately 10 degrees and hand position errors of approximately 6 mm, after Procrustes alignment. Results were consistent across camera viewpoints and robust to different methods for producing reference values from multiview video. This work extends monocular biomechanical analysis to detailed finger tracking, expanding access to quantitative characterization of hand movement from readily available video.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。