融合面部与身体动作信息，提升复杂场景下的说话人检测精度
ASDnB: Merging Face with Body Cues For Robust Active Speaker Detection
- 分阶段融合面部与身体特征，降低计算开销
- 在AVA-ActiveSpeaker等数据集上达顶尖性能
- 适合监控、野外等难以获取面部的场景使用
当前最先进的主动说话人检测（ASD）方法主要依赖音频和面部特征。本文提出新假设：身体动态也与‘说话’和‘倾听’高度相关，尤其在面部难以可靠获取的野外环境（如监控场景）中更具价值。我们提出ASDnB模型，通过在特征提取不同阶段融合面部与身体信息，实现端到端协同。该方法将3D卷积拆分为2D与1D以降低计算成本，同时采用自适应权重机制优化面部与身体特征的互补性。实验表明，ASDnB在基准数据集AVA-ActiveSpeaker、挑战性数据集WASD以及跨域设置（使用Columbia数据集）中均取得领先表现。该模型具备多场景适应能力，可作为鲁棒性语音-视觉检测的强基线（代码开源于https://github.com/Tiago-Roxo/ASDnB）。
原文摘要 · Abstract (English)
State-of-the-art Active Speaker Detection (ASD) approaches mainly use audio and facial features as input. However, the main hypothesis in this paper is that body dynamics is also highly correlated to "speaking" (and "listening") actions and should be particularly useful in wild conditions (e.g., surveillance settings), where face cannot be reliably accessed. We propose ASDnB, a model that singularly integrates face with body information by merging the inputs at different steps of feature extraction. Our approach splits 3D convolution into 2D and 1D to reduce computation cost without loss of performance, and is trained with adaptive weight feature importance for improved complement of face with body data. Our experiments show that ASDnB achieves state-of-the-art results in the benchmark dataset (AVA-ActiveSpeaker), in the challenging data of WASD, and in cross-domain settings using Columbia. This way, ASDnB can perform in multiple settings, which is positively regarded as a strong baseline for robust ASD models (code available at https://github.com/Tiago-Roxo/ASDnB).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。