用少量语料微调语音模型，让尼泊尔人能用方言语音操作手机支付。
SpeakPay: Domain-Adaptive LoRA Fine-Tuning of Whisper for Low-Resource Nepali Financial Speech Recognition
- 用LoRA微调Whisper模型，仅需100句语音就让识别错误率减半。
- 财务指令识别错误率从129.95%降至42.58%，交易成功率提升20倍。
- 专为视障用户设计，数据与代码全公开，适合低资源语音项目参考。
尼泊尔的移动支付应用多为图形界面，对视障用户不友好。本文提出语音优先的数字钱包SpeakPay，核心贡献是针对低资源金融语音识别的领域自适应研究。构建了包含403条尼泊尔语金融语音指令（含发送、充值、余额查询等）的数据集NepFinSpeech-403，涵盖237个唯一数值。采用LoRA微调Whisper large-v2模型，在测试集上将词错误率从零样本基线的129.95%降至42.58%，相对降低67.2%；德瓦纳加里数字识别准确率从0.0%提升至73.9%。词级指标低估实际影响：领域适配使交易成功率达33.33%，较1.67%提升约20倍。个体语音层面差异显著（符号检验，p < 10^-17），各类指令均获提升。数据效率分析表明，仅需100条领域语料即可使零样本WER减半，性能在约300例后趋于稳定。错误分析发现系统性数值混淆模式（如零插入/删除、前缀幻觉）是交易失败主因。训练模型已部署为公开可用的语音网页应用。所有代码、数据集、模型权重及论文均开源于https://github.com/subedibiraj/speakpay。
原文摘要 · Abstract (English)
Mobile payment applications in Nepal are graphically mediated and largely inaccessible to visually impaired users. This paper presents SpeakPay, a voice-first digital wallet, and documents the central technical contribution: a controlled study of domain adaptation for low-resource financial speech recognition. We introduce NepFinSpeech-403, a 403-utterance dataset of Nepali financial voice commands (send, load, and balance operations spanning 237 unique numerals), and fine-tune Whisper large-v2 with LoRA. On the held-out test set, the domain-adapted model reduces Word Error Rate from 129.95% (zero-shot baseline) to 42.58% --- a 67.2% relative reduction --- and improves Devanagari numeral recognition accuracy from 0.0% to 73.9%. We find that word-level metrics understate the practical task-level impact: domain adaptation improves the Transaction Success Rate from 1.67% to 33.33%, a roughly 20x gain. The improvement is consistent at the individual-utterance level (sign test, $p < 10^{-17}$) and across all command types. A data efficiency analysis shows that as few as 100 domain-specific utterances are sufficient to halve the zero-shot WER, with performance plateauing around 300 examples. Error analysis reveals systematic numeral confusion patterns (zero insertion/deletion, prefix hallucination) that account for the majority of remaining transaction failures. The trained system is deployed as a publicly accessible voice-first web application. All code, dataset, model weights, and this paper are released at https://github.com/subedibiraj/speakpay.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。