用大模型反馈优化语音识别，提升生僻词识别准确率
Customizing Speech Recognition Model with Large Language Model Feedback
- 用大模型作评分器，给语音识别结果打分
- 在无标签数据上训练，实体词错误率降21%
- 适合需要高精度领域语音识别的场景
自动语音识别（ASR）系统在通用转录任务中表现良好，但在识别罕见命名实体和应对领域差异方面仍存在困难。相比之下，基于海量互联网数据训练的大语言模型（LLM）在多种领域中更具适应性。本文提出一种基于强化学习的无监督领域自适应方法，利用未标注数据，通过大模型反馈提升转录质量，尤其改善因领域不匹配导致的命名实体识别问题。给定上下文信息，框架采用大模型作为奖励模型，对ASR模型的假设输出进行评分，这些评分作为强化学习的奖励信号，用于微调ASR模型。实验表明，该方法在实体词错误率上比传统自训练方法降低21%。
原文摘要 · Abstract (English)
Automatic speech recognition (ASR) systems have achieved strong performance on general transcription tasks. However, they continue to struggle with recognizing rare named entities and adapting to domain mismatches. In contrast, large language models (LLMs), trained on massive internet-scale datasets, are often more effective across a wide range of domains. In this work, we propose a reinforcement learning based approach for unsupervised domain adaptation, leveraging unlabeled data to enhance transcription quality, particularly the named entities affected by domain mismatch, through feedback from a LLM. Given contextual information, our framework employs a LLM as the reward model to score the hypotheses from the ASR model. These scores serve as reward signals to fine-tune the ASR model via reinforcement learning. Our method achieves a 21\% improvement on entity word error rate over conventional self-training methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。