微调Whisper模型提升阿萨姆语语音识别准确率,适合资源有限环境使用。
Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models

- 基于Common Voice数据集,对Whisper模型进行受控微调。
- 词错误率降至43.17%,相对基线提升78.26%。
- 适配低资源场景,训练效率高,适合边缘设备部署。
针对阿萨姆语等形态丰富但标注数据稀缺的低资源语言,构建自动语音识别系统面临挑战。预训练的Whisper模型在阿萨姆语任务上表现不佳。本文提出一种基于Mozilla Common Voice 24.0-Assamese语料库的受控微调方案,采用混合精度训练与梯度累积技术,在Tesla T4 GPU上实现硬件优化训练。微调模型显著优于零样本基线,词错误率(WER)、字符错误率(CER)、匹配错误率(MER)和词信息损失(WIL)分别为43.17%、13.18%、43%和64.81%,相对改进率达78.26%、93.10%、57.0%和35.19%。语义评估显示,BLEU和METEOR得分分别达30.81和0.5262。同时,幻觉率降低96.70%,实时因子(RTF)提升32.38%。
原文摘要 · Abstract (English)
Developing Automatic Speech Recognition (ASR) for morphologically rich, low-resource languages such as Assamese is challenging due to insufficient annotated speech data. The pretrained Whisper model performs poorly on Assamese speech recognition tasks. This paper presents a controlled, fine-tuned Whisper-based Assamese ASR system trained on the Mozilla Common Voice 24.0-Assamese corpus. A hardware-aware optimized training pipeline is implemented for resource-constrained environments, employing mixed-precision training and gradient accumulation on Tesla 4 Graphics Processing Units (T4 GPUs). The proposed fine-tuned model significantly outperformed the Zero-shot baseline, yielding Word Error Rate (WER), Character Error Rate (CER), Match Error Rate (MER), and Word Infomation Loss (WIL) of 43.17\%, 13.18\%, 43\%, and 64.81\%, respectively, achieving significant relative improvements of 78.26\%, 93.10\%, 57.0\%, and 35.19\% over the baseline. Semantic evaluation of the fine-tuned model also demonstrates notable improvement over a zero baseline, attaining Bilingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit ORdering (METEOR) scores of 30.81 and 0.5262, respectively. Additionally, the predicted hallucination rate and Real-Time Factor (RTF) are substantially improved by 96.70\% and 32.38\%, compared to the zero-shot baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。