对比两种语音编码器在42种印度语言识别中的表现,发现细调后的Whisper更准,而冻结的FastConformer泛化能力更强。
A Comparative Study of Pre-trained Speech Encoders and Training Objectives for Large-Scale Indic Spoken Language Identification

- 用Whisper和FastConformer结合线性分类器做大规模印地语语音识别
- 冻结的FastConformer在跨语料库测试中宏准确率超90%,优于Whisper
- 层级软标签损失(HSM)效果最佳,尤其提升跨域泛化性能
由于印度语言数量众多、相关方言间语音重叠严重,且许多低资源语言标注数据稀缺,印地语口语识别(LID)极具挑战。本文系统比较了两种预训练语音编码器——Whisper与FastConformer——搭配线性分类器,在涵盖42种语言、四大语系的大型印度语种语音识别任务中的表现。模型在Vaani数据集上训练,并在跨语料库设置下于Vaani-Test(保留)、FLEURS与Kathbath上评估,考察领域泛化能力。冻结状态下的FastConformer在未进行任务适配的情况下,于FLEURS与Kathbath上实现超过90%的宏准确率,显著优于Whisper;而微调后的Whisper则在本域表现更优。三种训练目标中,层级软标签(HSM)对两类编码器均持续领先,尤其在跨域测试集上提升显著;而监督对比损失(CE + supCon)反而削弱了FastConformer的跨域泛化能力,表明其易过拟合至本域特征。按语系分析显示,中印雅利安语支最难区分,其中印地语–乌尔都语及萨德里–恰蒂斯加尔语–苏尔古贾语群为最常见混淆对。
原文摘要 · Abstract (English)
Spoken language identification (LID) for Indian languages is a challenging problem due to the large number of languages, significant phonetic overlap among related varieties, and the scarcity of labeled data for many low-resource languages. In this work, we present a systematic comparative study of two pre-trained speech encoders -- Whisper and FastConformer -- combined with a linear classifier for large-scale Indic LID spanning 42 languages across four linguistic families. We evaluate both encoders in frozen (linear probing) and fine-tuned settings, and compare three training objectives: cross-entropy (CE), supervised contrastive loss with cross entropy (CE + supCon), and hierarchical softmax (HSM). Models are trained on the Vaani dataset and evaluated in a cross-corpus setting on Vaani-Test (held-out), FLEURS, and Kathbath, providing insights into domain generalization. The frozen FastConformer encoder achieves over 90\% macro accuracy on FLEURS and Kathbath without any task-specific adaptation, substantially outperforming Whisper on out-of-domain benchmarks, while fine-tuned Whisper yields stronger in-domain performance. HSM consistently outperforms CE and CE+SupCon for both encoders across all benchmarks, with the largest gains on out-of-domain test sets. CE+SupCon degrades FastConformer's cross-corpus generalization, suggesting that the contrastive objective over-specializes representations to in-domain conditions. Per-family analysis shows that Central Indo-Aryan varieties are the hardest to discriminate, with Hindi--Urdu and the Sadri--Chhattisgarhi--Surgujia cluster being the dominant confusion pairs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。