arXiv:2606.14321cs.SDcs.MM2026-06被引 1

无脸情况下识别说乌尔都语的多语言说话人,准确率达99.89%。

MaskedFOP: Polyglot Speaker Identification under Missing Visual Modality via Cascaded Graph Label Propagation

论文配图:MaskedFOP: Polyglot Speaker Identification under Missing Visual Modality via Cascaded Graph Label Propagation
图 1 · 摘自论文原文
  • 通过随机遮蔽人脸训练音频分支,使其独立具备判别能力。
  • 融合两组不同随机种子模型的音频特征,生成更鲁棒的512维表示。
  • 分阶段图标签传播+余弦最近中心点匹配,提升小样本场景表现。

我们提出MaskedFOP系统,用于在测试时完全缺失人脸模态且语音为训练中未见的乌尔都语条件下,进行闭集多语言说话人识别。系统集成三种互补机制:首先,基于FOP主干的模态丢弃双分支网络通过对每样本人脸进行随机遮蔽,强制音频分支发展独立判别能力;其次,两个在ECAPA-TDNN特征上以不同随机种子训练的MaskedFOP实例生成互补音频嵌入,其元素平均得到比单模型更稳健的512维表示;第三,采用两级级联推理:第一阶段通过融合图标签传播(GLP)优化多模态标签,第二阶段使用余弦最近中心点分配音频仅标签,将70个稀疏训练原型替换为约1,500个域内测试集中心点。提交至POLY-SIM 2026挑战赛,系统取得均值P-准确率0.9989,排名第一。消融实验表明级联种子是最大增益来源(提升超8个百分点)。代码已开源。

原文摘要 · Abstract (English)

We present MaskedFOP, a system for closed-set polyglot speaker identification under two simultaneous challenges: the face modality is entirely absent at test time, and speech comes from Urdu, a language unseen during face-supervised training. The system integrates three complementary mechanisms. First, a modality-dropout dual-head network built on the Fusion and Orthogonal Projection (FOP) backbone forces the audio branch to develop independent discriminative power via per-sample face masking, ensuring that the audio encoder remains capable when face is absent. Second, two MaskedFOP instances trained on Emphasized Channel Attention, Propagation, and Aggregation in Time Delay Neural Network (ECAPA-TDNN) features with different random seeds produce complementary audio embeddings whose element-wise average yields a more robust 512-dimensional representation than any single model. Third, a two-stage cascaded inference procedure first refines multimodal labels through a fused Graph Label Propagation (GLP) pass (Stage 1), then assigns audio-only labels by cosine nearest-centroid (Stage 2), replacing the 70 sparse training prototypes with ~1,500 in-domain test-set centroids from Stage 1. Submitted to the POLY-SIM 2026 Grand Challenge, the system achieves a mean P-accuracy of 0.9989, placing first among all submissions evaluated on the challenge server. An ablation identifies cascaded seeding as the single largest gain (>8 pp on P4/P6). The code is available at https://github.com/Ayoub-Elkhouzari/POLY-SIM2026.

说话人识别多模态无脸识别乌尔都语

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。