提出可复现的医疗语音识别评估方法，精准捕捉关键错误。
MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognition
- 用固定术语表替代依赖NER模型的评估方式
- 在两个公开数据集上对多个模型进行评估，结果带95%置信区间
- 适合医疗语音识别研究者和需要高精度评估的团队
整体词错误率会掩盖临床关键错误：一份转录文本95%正确仍可能将一种药物误换为另一种。传统改进方法虽对医学实体加权，但几乎都依赖评估时的命名实体识别（NER）模型或云API，使评估分母成为版本化的黑箱。本文提出MedWER评估协议及开源工具，其分母为固定、许可证清洁的术语列表：从公开源数据投影出的19,373个药物、诊断、症状和损伤机制条目。该协议结合固定文本归一化器与短语感知的术语受限词错误率（MedWER），唯一版本化组件是精确版本的归一化依赖，并通过提交的黄金样本验证。术语表覆盖度经独立省级药品福利文件验证，匹配启发式方法基于真实实体跨度校准。在两个开放基准上，对Moonshine~base、Whisper~base.en和MedASR的基线模型使用发布工具进行评分，并报告了基于重采样每句得分的95%置信区间。
原文摘要 · Abstract (English)
Overall word error rate hides clinically critical errors: a transcript can be 95% correct and still swap one drug for another. The usual fix weights errors on medical entities, and almost always depends on an evaluation-time named-entity recognition (NER) model or cloud API, which makes the metric's denominator a versioned black box. We present MedWER, an evaluation protocol and open-source tool for medical ASR whose denominator is a fixed, license-clean term list: 19,373 drug, diagnosis, symptom, and injury-mechanism entries projected from public sources. The protocol couples a pinned text normalizer with a phrase-aware term-restricted WER, the MedWER, so the only versioned component is a normalizer dependency held at an exact release and checked against committed golden fixtures. Coverage is validated against an independent provincial drug-benefit file the list was not built from; the matching heuristic is calibrated against ground-truth entity spans. Baselines for Moonshine~base, Whisper~base.en, and MedASR on two open benchmarks are scored with the released tool and reported with 95% confidence intervals from resampled per-utterance scores.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。