对比编码器与生成式大模型在语音识别评估中的表现
Generative vs. Encoder Large Language Models for ASR Evaluation: A Comparative Study
- 用编码器和生成式大模型分别计算语义相似度
- 编码器模型在语义相关性上表现优异，生成模型在假设比较中更强
- 生成模型提升评估可解释性，适合需要分析错误的场景
自动语音识别（ASR）通常使用词错误率（WER）评估，但其无法有效反映语义相似性。基于嵌入的指标虽与人工判断更相关，但编码器型与生成型大语言模型（LLM）的作用仍不明确。本文对两类模型在ASR评估中的表现进行了对比研究。我们分析了BERTScore与SemDist在不同LLM、层及池化策略下的表现，发现两者在合理配置下均能与人工判断高度相关。针对生成型模型，考察了两种设置：通过提示进行成对假设选择，以及直接进行定性错误分类。结果表明，编码器型指标仍具竞争力，而生成型模型在假设比较中表现突出，并显著提升了评估的可解释性。
原文摘要 · Abstract (English)
Automatic Speech Recognition (ASR) is typically evaluated using Word Error Rate (WER), which poorly reflects semantic similarity. While embedding-based metrics correlate better with human judgments, the respective roles of encoder and decoder-based Large Language Models (LLMs) remain underexplored. This paper presents a comparative study of both families for ASR evaluation. We analyze BERTScore and SemDist across different LLMs, layers, and pooling strategies, showing that both metrics can achieve strong correlation with human judgments when properly configured. For decoder models, we investigate generative LLMs in two settings: pairwise hypothesis selection via prompting and direct qualitative error classification. Our results show that encoder-based metrics remain highly competitive, while generative LLMs perform strongly in hypothesis comparison and improve the interpretability of ASR evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。