让Whisper支持普什图语语音识别，通过微调显著降低错误率。
Fine-tuning Whisper for Pashto ASR: strategies and scale
- 用全量微调法在113小时数据上训练Whisper模型，效果最佳。
- 微调后普什图语识别错误率降至21.22%，远超其他方法。
- 适合需要低成本部署的低资源语言语音识别研究者。
普什图语未出现在Whisper的预训练语料中，尽管CommonVoice收录了其大量语音数据，导致默认模型对普什图语音输出阿拉伯、达里或乌尔都文字符，词错误率超过100%。本文在CommonVoice Pashto v20上对比四种微调策略：全量微调、LoRA（秩64）、冻结编码器（2/6层）以及多阶段乌尔都→普什图迁移。将全量微调扩展至whisper-small和whisper-large-v3-turbo在v24数据集（113小时）上。全量微调在CV20上实现21.22%的词错误率，优于LoRA（-33.36pp）、冻结编码器（-14.76pp）和迁移学习（-44.56pp）。冻结编码器在whisper-base（6层编码器）上表现下降，因层功能分离不成立且损失三分之一可训练参数。迁移学习因中间检查点不可靠、音系差异及训练不足而失败。在CV24上，whisper-small达24.89%（比whisper-base高2.24pp，参数量为3.3倍）；whisper-large-v3-turbo达23.37%（再降1.52pp）。收益递减表明whisper-small是113小时数据下的最优选择。在线增强带来7.25pp的错误率改善。错误分析显示主要问题为词尾后缀混淆（阳性-ay vs 阴性-a）及包含普什图特有辅音/ts/的卷舌音替换。微调模型与评估脚本已发布于HuggingFace。
原文摘要 · Abstract (English)
Pashto is absent from Whisper's pre-training corpus despite being one of CommonVoice's largest language collections, leaving off-the-shelf models unusable: all Whisper sizes output Arabic, Dari, or Urdu script on Pashto audio, achieving word error rates above 100%. We compare four fine-tuning strategies for whisper-base on CommonVoice Pashto v20: vanilla full fine-tuning, LoRA (rank 64), frozen-encoder (2/6 layers), and multistage Urdu-to-Pashto transfer. We extend vanilla fine-tuning to whisper-small and whisper-large-v3-turbo on CommonVoice Pashto v24 (113 hours). Vanilla fine-tuning achieves WER 21.22% on CV20, outperforming LoRA by 33.36 pp, frozen-encoder by 14.76 pp, and Urdu transfer by 44.56 pp. Frozen-encoder fine-tuning degrades performance on whisper-base (6 encoder layers): layer-function separation does not hold at this depth, and freezing removes a third of trainable capacity. Urdu-to-Pashto transfer fails due to an unverified intermediate checkpoint, phonological mismatch, and insufficient training. On CV24, whisper-small achieves WER 24.89% (2.24 pp over whisper-base at 3.3x parameters); whisper-large-v3-turbo achieves 23.37% (a further 1.52 pp). Diminishing returns indicate whisper-small is the practical optimum at 113 hours. Online augmentation provides 7.25 pp WER benefit over matched training. Error analysis identifies word-final suffix confusion (masculine -ay vs. feminine -a) and retroflex substitutions involving the Pashto-unique consonant /ts/ as dominant failure modes. Fine-tuned checkpoints and evaluation scripts are released on HuggingFace.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。