解决多文字拼写导致的语音识别错误率虚高问题
SN-WER: Script-Normalized WER for Multi-Script Indic ASR Evaluation

- 将参考文本和识别结果统一转为标准文字再计算错误率
- 在5种印地语系语言上降低12%的虚假误差差距
- 适合用于多语言搜索与大模型下游任务评估
词错误率(WER)是语音识别的主要评估指标,但在参考文本与识别输出使用不同文字编码相同词语时,会过度估计错误。这种情况在多语言场景中常见,例如模型输出罗马化文本。本文提出无需训练、仅用于评估的脚本归一化词错误率(SN-WER),先将参考与假设文本统一转写为特定语言的标准文字,再计算WER。我们在5种印地语系语言、2个数据集和3个ASR模型上验证了该方法。在精选的FLEURS数据上,SN-WER使被夸大的模型差距减少高达12%;在更嘈杂的Common Voice数据上,改善较小或不一致,表明存在真实识别弱点而非仅文字差异。受控测试显示,人工罗马化导致的错误率虚增被降低了67%,而词汇替换控制显示对语义错误敏感度与传统WER接近,ΔSN-WER / ΔWER ≈ 1.09。SN-WER对转写器选择、归一化方式均鲁棒,所涉印地语系场景中词碰撞率低于0.1%。我们主张在报告WER和CER的同时,加入SN-WER作为脚本无关评估的辅助指标,尤其适用于需文本下游处理的检索、索引或多语言大模型流程。
原文摘要 · Abstract (English)
Word Error Rate (WER) is the dominant metric for automatic speech recognition (ASR), but it can overestimate errors when references and hypotheses encode the same words in different scripts. This issue is common in multilingual settings where ASR models may emit romanized text. We propose Script-Normalized WER (SN-WER), a training-free, evaluation-only scoring method that transliterates both reference and hypothesis text into a language-specific canonical script before computing WER. We evaluate SN-WER on 5 Indic languages, 2 datasets, and 3 ASR models. On curated FLEURS data, SN-WER reduces inflated model gaps by up to 12%, while on noisier Common Voice data the reductions are smaller or inconsistent, indicating genuine recognition weaknesses rather than only script mismatch. Controlled stress tests show a 67% attenuation of artificial romanization-induced WER inflation, while lexical-substitution controls show near-identical sensitivity to semantic errors, with Delta SN-WER / Delta WER approximately 1.09. SN-WER is robust to transliterator choice, normalization changes, and shows low token-collision rates below 0.1% in the evaluated Indic setting. We argue that SN-WER should be reported alongside WER and CER as a companion metric for script-insensitive ASR evaluation, especially when transcripts feed downstream search, indexing, or multilingual LLM pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。