用语音和视频联合判断手术反馈是否有效，提升培训质量。
Multi-Modal Self-Supervised Learning for Surgical Feedback Effectiveness Assessment
- 融合训练者语音与手术视频，预测反馈效果。
- 多模态结合使准确率提升6.6%，AUROC达0.70±0.02。
- 自监督微调提升视频表征，适合医学教育研究者。
手术训练中，导师实时反馈对防止错误、促进技能长期掌握至关重要。准确预测反馈是否引发学员行为改变，是改进手术教学的关键。但依赖人工标注评估反馈效果成本高且易受偏见影响，亟需自动化、可扩展、客观的方法。该任务难点在于需同时理解导师的口头反馈和实时手术场景的视觉信息。为此，我们提出一种融合转录语音与对应手术视频的方法，用于预测反馈有效性。结果表明，单独使用语音或视频均能预测行为变化，两者结合使AUROC达到0.70±0.02，准确率最高提升6.6%。此外，引入自监督微调策略增强手术视频表征学习，具可扩展性并进一步提升性能。研究证明多模态学习在自动化评估手术反馈方面的潜力。
原文摘要 · Abstract (English)
During surgical training, real-time feedback from trainers to trainees is important for preventing errors and enhancing long-term skill acquisition. Accurately predicting the effectiveness of this feedback, specifically whether it leads to a change in trainee behavior, is crucial for developing methods for improving surgical training and education. However, relying on human annotations to assess feedback effectiveness is laborious and prone to biases, underscoring the need for an automated, scalable, and objective method. Creating such an automated system poses challenges, as it requires an understanding of both the verbal feedback delivered by the trainer and the visual context of the real-time surgical scene. To address this, we propose a method that integrates information from transcribed verbal feedback and corresponding surgical video to predict feedback effectiveness. Our findings show that both transcribed feedback and surgical video are individually predictive of trainee behavior changes, and their combination achieves an AUROC of 0.70+/-0.02, improving prediction accuracy by up to 6.6%. Additionally, we introduce self-supervised fine-tuning as a strategy for enhancing surgical video representation learning, which is scalable and further enhances prediction performance. Our results demonstrate the potential of multi-modal learning to advance the automated assessment of surgical feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。