剖析Whisper模型在不同语言资源下的子词解码差异，揭示语音识别公平性隐忧
Beyond WER: Probing Whisper's Sub-token Decoder Across Diverse Language Resource Levels
- 追踪beam search路径，分析各语言子词猜测与置信度
- 高资源语言正确词更易排第一，信心更高，熵更低
- 低资源语言解码表现差，且子词使用模式受语言类型影响
尽管大型多语言自动语音识别（ASR）模型表现卓越，但其端到端流水线内部机制，尤其是跨语言的公平性与有效性仍缺乏深入研究。本文对Whisper的多语言解码器进行了细粒度分析，考察了在不同资源水平语言上的子词推测过程。通过追踪beam search路径，捕捉子词候选及其概率。结果表明，高资源语言中正确子词更可能位居前列，具有更高置信度、更低预测熵和更多样化的备选方案；而低资源语言在这些指标上表现较差，且子词使用呈现特定聚类模式，部分受语言类型影响（主成分分析与t-SNE验证）。该子词探测揭示了被整体错误率掩盖的系统性解码差异，提示应针对语音技术发展不平衡问题采取针对性改进措施。
原文摘要 · Abstract (English)
While large multilingual automatic speech recognition (ASR) models achieve remarkable performance, the internal mechanisms of the end-to-end pipeline, particularly concerning fairness and efficacy across languages, remain underexplored. This paper introduces a fine-grained analysis of Whisper's multilingual decoder, examining its sub-token hypotheses during transcription across languages with various resource levels. Our method traces the beam search path, capturing sub-token guesses and their associated probabilities. Results reveal that higher resource languages benefit from higher likelihood of the correct token being top-ranked, greater confidence, lower predictive entropy, and more diverse alternative candidates. Lower resource languages fare worse on these metrics, but also exhibit distinct clustering patterns in sub-token usage sometimes influenced by typology in our PCA and t-SNE analysis. This sub-token probing uncovers systematic decoding disparities masked by aggregate error rates and points towards targeted interventions to ameliorate the imbalanced development of speech technology.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。