用大模型评估语音识别，比传统方法更贴近人类理解。
Evaluation of Automatic Speech Recognition Using Generative Large Language Models
- 用生成式大模型比较识别结果优劣，判断哪个更合理。
- 在HATS数据集上，大模型与人工判断一致率达92%~94%，远超传统方法。
- 能解释错误类型，适合需要可解释性评估的场景。
传统语音识别评估依赖词错误率（WER），该指标对语义不敏感。基于嵌入的语义度量更贴近人类感知，但解码器型大语言模型（LLM）在此任务中尚未被充分探索。本文通过三种方式评估其有效性：(1) 在两个候选输出间选择最优结果；(2) 使用生成式嵌入计算语义距离；(3) 定性分类错误。在HATS数据集上，最佳大模型与人工标注者达成92%~94%的一致性，显著优于63%的WER表现，也超越现有语义度量。解码器型大模型生成的嵌入性能接近编码器模型。结果表明，大模型为可解释且语义导向的语音识别评估提供了有前景的方向。
原文摘要 · Abstract (English)
Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92--94\% agreement with human annotators for hypothesis selection, compared to 63\% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。