提出分层门控融合架构,提升视频中说话人检测的准确率
GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection

- 采用分层门控机制,在多层Transformer中渐进融合音视频特征
- 在多个基准上达到新最高性能,最高提升9.4% mAP
- 适合需要高精度说话人定位的多模态应用场景
主动说话人检测(ASD)旨在识别视频每一帧中正在讲话的人。现有先进方法多依赖晚期融合来结合视觉与音频特征,但晚期融合难以捕捉细粒度跨模态交互,这在非受限场景下会影响鲁棒性。本文提出GateFusion,一种结合强预训练单模态编码器与分层门控融合解码器(HiGate)的新架构。HiGate通过可学习的双模态条件门控,在Transformer主干网络的多个层级自适应地将一模态上下文特征注入另一模态,实现渐进式多深度融合。为增强多模态学习,提出两种辅助目标:掩码对齐损失(MAL)以对齐单模态输出与多模态预测,过正惩罚(OPP)用于抑制仅视频引发的虚假激活。GateFusion在多个挑战性ASD基准上取得新最佳表现:Ego4D-ASD、UniTalk和WASD的mAP分别达77.8%(+9.4%)、86.1%(+2.9%)和96.1%(+0.5%),并在AVA-ActiveSpeaker上表现良好。域外实验验证了模型泛化能力,全面消融实验表明各组件具有互补优势。
原文摘要 · Abstract (English)
Active Speaker Detection (ASD) aims to identify who is currently speaking in each frame of a video. Most state-of-the-art approaches rely on late fusion to combine visual and audio features, but late fusion often fails to capture fine-grained cross-modal interactions, which can be critical for robust performance in unconstrained scenarios. In this paper, we introduce GateFusion, a novel architecture that combines strong pretrained unimodal encoders with a Hierarchical Gated Fusion Decoder (HiGate). HiGate enables progressive, multi-depth fusion by adaptively injecting contextual features from one modality into the other at multiple layers of the Transformer backbone, guided by learnable, bimodally-conditioned gates. To further strengthen multimodal learning, we propose two auxiliary objectives: Masked Alignment Loss (MAL) to align unimodal outputs with multimodal predictions, and Over-Positive Penalty (OPP) to suppress spurious video-only activations. GateFusion establishes new state-of-the-art results on several challenging ASD benchmarks, achieving 77.8% mAP (+9.4%), 86.1% mAP (+2.9%), and 96.1% mAP (+0.5%) on Ego4D-ASD, UniTalk, and WASD benchmarks, respectively, and delivering competitive performance on AVA-ActiveSpeaker. Out-of-domain experiments demonstrate the generalization of our model, while comprehensive ablations show the complementary benefits of each component.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。