用身体姿态提升语音说话人检测准确率,尤其在遮挡和噪音下表现更好。
FabuLight-ASD: Unveiling Speech Activity via Body Language
- 融合面部、音频与人体骨骼姿态信息,构建轻量级多模态检测模型
- 在WASD数据集上达到94.3%的mAP,优于原模型93.7%
- 仅增加27.3%参数和少量计算量,适合实际部署
在多模态环境中,主动说话人检测(ASD)对视频会议、人机交互等应用至关重要。本文提出FabuLight-ASD,一种结合面部、音频与人体姿态信息的先进ASD模型。该模型基于Light-ASD框架,引入通过骨骼图表示的人体姿态数据,有效降低计算开销。利用标注可靠的Wilder Active Speaker Detection(WASD)数据集,实验表明,FabuLight-ASD在真实场景中表现出色,整体平均精度(mAP)达94.3%,优于原Light-ASD模型的93.7%。在语音障碍、面部遮挡及背景人声等挑战性场景中,姿态信息带来显著性能提升。效率分析显示,模型参数仅增加27.3%,乘加操作最多上升2.4%,验证了其高效性与实用性。代码与模型权重已开源。
原文摘要 · Abstract (English)
Active speaker detection (ASD) in multimodal environments is crucial for various applications, from video conferencing to human-robot interaction. This paper introduces FabuLight-ASD, an advanced ASD model that integrates facial, audio, and body pose information to enhance detection accuracy and robustness. Our model builds upon the existing Light-ASD framework by incorporating human pose data, represented through skeleton graphs, which minimises computational overhead. Using the Wilder Active Speaker Detection (WASD) dataset, renowned for reliable face and body bounding box annotations, we demonstrate FabuLight-ASD's effectiveness in real-world scenarios. Achieving an overall mean average precision (mAP) of 94.3%, FabuLight-ASD outperforms Light-ASD, which has an overall mAP of 93.7% across various challenging scenarios. The incorporation of body pose information shows a particularly advantageous impact, with notable improvements in mAP observed in scenarios with speech impairment, face occlusion, and human voice background noise. Furthermore, efficiency analysis indicates only a modest increase in parameter count (27.3%) and multiply-accumulate operations (up to 2.4%), underscoring the model's efficiency and feasibility. These findings validate the efficacy of FabuLight-ASD in enhancing ASD performance through the integration of body pose data. FabuLight-ASD's code and model weights are available at https://github.com/knowledgetechnologyuhh/FabuLight-ASD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。