提出多视角姿态感知模型，实时判断技能动作执行质量
SkillSpotter: Pose-Aware Multi-View Skilled Action Detection and Grading in Ego-Exo Videos
- 融合姿态与视觉信息，通过三模块联合检测与评分
- 在Ego-Exo4D上提升准确率76%，平衡精度达60.40%
- 适用于体育、烹饪等场景的实时教学反馈系统
为在体育、烹饪或音乐等领域实现基于增强现实眼镜或固定摄像头的个性化实时指导，系统不仅需识别动作，还需评估其执行质量。在自拍-外拍视频场景中，这要求同时检测技能动作并分类为正确或需改进，该任务由Ego-Exo4D的熟练度演示基准正式定义。我们首先将七种先进时序动作检测架构适配此任务，扩展评估协议以解耦检测与评分，并发现现有方法评分近乎随机。随后提出SkillSpotter，一种姿态感知的多视角架构，通过三个专用模块联合完成动作检测与评分：(1) 自适应时间抑制以处理不同活动间技能动作密度差异；(2) 门控3D人体姿态融合，利用身体运动学作为视觉特征的补充信号；(3) 双向跨视角注意力，有效结合第一人称与第三人称视角。SkillSpotter将类特定mAP从12.40提升至21.82（+76%），平衡精度从55.99%提升至60.40%，优于最佳基线。其模块可迁移至其他时序动作检测模型并持续增益，且在HoloAssist上也表现良好。代码已开源。
原文摘要 · Abstract (English)
To enable personalized, real-time coaching using Augmented Reality glasses or fixed camera setups in domains such as sports, cooking, or music, a system must understand not just what a person does, but how well they execute an activity. In an ego-exo video setting, this requires simultaneously detecting individual skilled actions and classifying each as correct or needing improvement, which Ego-Exo4D's proficiency demonstration benchmark formalized. We first adapt seven state-of-the-art temporal action detection architectures to this task, extend the evaluation protocol to disentangle detection from grading, and show that existing methods grade near-randomly. We then introduce SkillSpotter, a pose-aware multi-view architecture that jointly detects and grades skilled actions through three task-specific modules: (1) adaptive temporal suppression to handle the varying density of skilled actions across diverse activities, (2) gated 3D body pose fusion to leverage body kinematics as a complementary signal to visual features, and (3) bidirectional cross-view attention to combine ego and exo views effectively. SkillSpotter improves class-specific mAP from 12.40 to 21.82 (+76%) and balanced accuracy from 55.99% to 60.40% over the best baseline. SkillSpotter's modules transfer to other temporal action detection models with consistent gains, and our method generalizes beyond Ego-Exo4D to HoloAssist. Code: https://github.com/eth-siplab/SkillSpotter
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。