改进Whisper解码器，提升德拉威语等低资源语言的识别准确率
Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languages
- 引入加权注意力与自条件机制，平衡语言与声学信息
- 在低资源和黏着语上实现稳定词错误率降低
- 适合研究低资源语言语音识别的学者与工程师
多语言语音识别模型Whisper在高资源语言上表现良好，但在德拉威语上的词错误率（WER）显著高于印欧语系语言。通过语言学与数据集分析发现，德拉威语词汇更长、词汇多样性更高、重复率更低，导致标记分布稀疏，常出现字符级替换错误。基线微调进一步揭示解码器中自注意力（语言上下文）与交叉注意力（声学线索）存在失衡。尽管合成重复实验显示潜在收益，但实际不可行。为此，我们提出两种解码器级改进：加权注意力可自适应平衡注意力来源，自条件机制则将中间预测结果回传以增强标记一致性。实验表明，该方法对低资源及黏着语种实现了稳定的WER下降。
原文摘要 · Abstract (English)
Multilingual ASR models such as Whisper perform well on high-resource languages but exhibit substantially higher Word Error Rates (WER) for Dravidian languages compared to Indo-Aryan ones. Through linguistic and dataset analysis, we show that Dravidian languages have longer words, higher vocabulary diversity, and lower repetition, resulting in sparse token distributions and frequent character-level substitution errors. Baseline fine-tuning further reveals decoder imbalance between self-attention (linguistic context) and cross-attention (acoustic cues). Although synthetic token-repetition experiments indicate potential gains, they are impractical. Motivated by these observations, we introduce two decoder-level enhancements: Weighted-Attention, which adaptively balances attention sources, and Self-Conditioning, which reinjects intermediate predictions to improve token consistency. Experiments demonstrate consistent WER reductions for low-resource and agglutinative languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。