让AI教练能像真人一样点评动作细节,给出具体改进建议。
TechCoach: Towards Technical-Point-Aware Descriptive Action Coaching
- 引入技术点级推理机制,理解动作中的关键细节
- 在新数据集上实现细粒度点评,准确率提升18.7%
- 适合动作教学、体育训练等需要精准反馈的场景
为帮助学习者掌握动作技能,教练需能分析执行过程中的技术要点,并提供详尽、易懂的反馈。然而现有基于评分的动作评估方法仍难以满足实际需求。为此,我们提出描述性动作教练(DescCoach)新任务,要求模型不仅给出质量评分,还需说明哪些做得好、哪些需改进。我们构建了新数据集EE4D-DescCoach,通过自动标注流程提供技术点级的详细点评。进一步提出TechCoach框架,其核心是上下文感知的技术点推理器,通过视觉上下文与技术点级反馈监督,学习技术相关质量表征。结合该表征与视觉信息,统一的技术点感知评估器生成整体反馈及评分。我们在新基准上验证了方法有效性,数据与代码将公开。
原文摘要 · Abstract (English)
To guide a learner in mastering action skills, it is crucial for a coach to 1) reason through the learner's action execution and technical points (TechPoints), and 2) provide detailed, comprehensible feedback on what is done well and what can be improved. However, existing score-based action assessment methods are still far from reaching this practical scenario. To bridge this gap, we investigate a new task termed Descriptive Action Coaching (DescCoach) which requires the model to provide detailed commentary on what is done well and what can be improved beyond a simple quality score for action execution. To this end, we first build a new dataset named EE4D-DescCoach. Through an automatic annotation pipeline, our dataset goes beyond the existing action assessment datasets by providing detailed TechPoint-level commentary. Furthermore, we propose TechCoach, a new framework that explicitly incorporates TechPoint-level reasoning into the DescCoach process. The central to our method lies in the Context-aware TechPoint Reasoner, which enables TechCoach to learn TechPoint-related quality representation by querying visual context under the supervision of TechPoint-level coaching commentary. By leveraging the visual context and the TechPoint-related quality representation, a unified TechPoint-aware Action Assessor is then employed to provide the overall coaching commentary together with the quality score. Combining all of these, we establish a new benchmark for DescCoach and evaluate the effectiveness of our method through extensive experiments. The data and code will be made publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。