同时追踪动物与人类并估计关键点姿态,效率更高更通用。
STEP: Simultaneous Tracking and Estimation of Pose for Animals and Humans
- 用Transformer模型实现姿态与追踪同步预测,无需每帧重新检测目标。
- 引入高斯图软预测与偏移图回归适配器,摆脱对关键点真值的依赖。
- 适用于多物种行为分析,尤其适合长视频中高效连续追踪场景。
我们提出STEP框架,利用基于Transformer的判别模型实现跨多种动物和人类的姿态估计与同步追踪。受人脑在形式与运动处理区域分工但能协同完成定位与姿态识别的启发,传统判别模型通常需预设目标状态来确定权重,我们通过高斯图软预测(GMSP)与偏移图回归适配器(OMRA)模块解决了这一问题,无需输入关键点真值即可训练与推理。方法从视频首帧已知目标状态出发,后续帧无缝追踪目标并输出重要解剖关键点。相比主流自上而下方法,该方案不依赖每帧目标检测,显著提升推理效率,拓展了应用潜力。我们在涵盖多种物种的数据集上训练与验证,实验表明性能优于现有方法,为动作识别与行为分析等任务开辟新路径。
原文摘要 · Abstract (English)
We introduce STEP, a novel framework utilizing Transformer-based discriminative model prediction for simultaneous tracking and estimation of pose across diverse animal species and humans. We are inspired by the fact that the human brain exploits spatiotemporal continuity and performs concurrent localization and pose estimation despite the specialization of brain areas for form and motion processing. Traditional discriminative models typically require predefined target states for determining model weights, a challenge we address through Gaussian Map Soft Prediction (GMSP) and Offset Map Regression Adapter (OMRA) Modules. These modules remove the necessity of keypoint target states as input, streamlining the process. Our method starts with a known target state in the initial frame of a given video sequence. It then seamlessly tracks the target and estimates keypoints of anatomical importance as output for subsequent frames. Unlike prevalent top-down pose estimation methods, our approach doesn't rely on per-frame target detections due to its tracking capability. This facilitates a significant advancement in inference efficiency and potential applications. We train and validate our approach on datasets encompassing diverse species. Our experiments demonstrate superior results compared to existing methods, opening doors to various applications, including but not limited to action recognition and behavioral analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。