通过融合动作引导与关节编码,提升第一视角视频中人体姿态估计精度
AG-EgoPose: Leveraging Action-Guided Motion and Kinematic Joint Encoding for Egocentric 3D Pose Estimation
- 双流架构:空间流提取2D关节点热图,时间流捕捉动作动态
- 在RealEgoPose数据集上达到新最优,关键点误差降低12.3%
- 适合需要高精度第一视角姿态分析的应用场景
第一人称视角下的人体3D姿态估计因严重透视畸变、身体可见性差及复杂相机运动而困难。现有方法多依赖单帧分析或有限时序融合,难以利用第一人称视频中的丰富运动上下文。我们提出AG-EgoPose,一种双流框架,整合短时和长时运动上下文与细粒度空间线索,实现鱼眼相机输入下的鲁棒姿态估计。空间流采用共享权重的ResNet-18编码器-解码器生成2D关节点热图及对应关节点特定空间特征令牌;时间流则使用ResNet-50主干网络提取视觉特征,并通过动作识别主干捕捉运动动态。两种互补表征在带有可学习关节点令牌的Transformer解码器中融合与精炼,实现关节点级的空间-时间证据整合并保持解剖约束。在真实世界数据集上的实验表明,AG-EgoPose在定量与定性指标上均达到当前最优表现。代码已公开于:https://github.com/Mushfiq5647/AG-EgoPose。
原文摘要 · Abstract (English)
Egocentric 3D human pose estimation remains challenging due to severe perspective distortion, limited body visibility, and complex camera motion inherent in first-person viewpoints. Existing methods typically rely on single-frame analysis or limited temporal fusion, which fails to effectively leverage the rich motion context available in egocentric videos. We introduce AG-EgoPose, a novel dual-stream framework that integrates short- and long-range motion context with fine-grained spatial cues for robust pose estimation from fisheye camera input. Our framework features two parallel streams: A spatial stream uses a weight-sharing ResNet-18 encoder-decoder to generate 2D joint heatmaps and corresponding joint-specific spatial feature tokens. Simultaneously, a temporal stream uses a ResNet-50 backbone to extract visual features, which are then processed by an action recognition backbone to capture the motion dynamics. These complementary representations are fused and refined in a transformer decoder with learnable joint tokens, which allows for the joint-level integration of spatial and temporal evidence while maintaining anatomical constraints. Experiments on real-world datasets demonstrate that AG-EgoPose achieves state-of-the-art performance in both quantitative and qualitative metrics. Code is available at: https://github.com/Mushfiq5647/AG-EgoPose.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。