用Whisper模型直接生成语音识别置信度，效果优于传统方法。
Adopting Whisper for Confidence Estimation
- 直接微调Whisper模型输出词级置信分数，端到端实现。
- Whisper-tiny在域内表现相当，8个域外数据集上超越基线。
- Whisper-large全数据集显著领先，适合追求高精度场景。
近年来，词级置信度估计研究多依赖于轻量级的置信度估计模块（CEMs），其基于自动语音识别（ASR）输出的手工特征。本文提出一种新方法，直接利用ASR模型Whisper生成词级置信度分数。具体而言，我们对Whisper模型进行微调，使其在给定音频输入和对应转录文本时输出标量置信度分数。实验表明，微调后的Whisper-tiny模型（与强基线CEM大小相当）在域内数据集上表现相当，且在8个域外数据集上超越基线；而微调后的Whisper-large模型在所有数据集上均显著优于基线。
原文摘要 · Abstract (English)
Recent research on word-level confidence estimation for speech recognition systems has primarily focused on lightweight models known as Confidence Estimation Modules (CEMs), which rely on hand-engineered features derived from Automatic Speech Recognition (ASR) outputs. In contrast, we propose a novel end-to-end approach that leverages the ASR model itself (Whisper) to generate word-level confidence scores. Specifically, we introduce a method in which the Whisper model is fine-tuned to produce scalar confidence scores given an audio input and its corresponding hypothesis transcript. Our experiments demonstrate that the fine-tuned Whisper-tiny model, comparable in size to a strong CEM baseline, achieves similar performance on the in-domain dataset and surpasses the CEM baseline on eight out-of-domain datasets, whereas the fine-tuned Whisper-large model consistently outperforms the CEM baseline by a substantial margin across all datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。