用注意力权重估算术语概率，提升语音识别中专有名词的准确率。
Attention2Probability: Attention-Driven Terminology Probability Estimation for Robust Speech-to-Text System
- 通过语音与术语的交叉注意力权重生成术语出现概率。
- 中文和英文术语召回率分别达92.57%和86.83%，每查询延迟仅8.71ms。
- 轻量高效，适合在语音大模型中插入术语增强，特别适合医疗、法律等专业场景。
语音大语言模型（SLMs）在通用领域语音识别与翻译方面取得进展，但对领域专有术语或新词的生成仍存挑战。为此，本文提出Attention2Probability：一种轻量、灵活且精准的术语概率估计方法，将语音与术语间的交叉注意力权重转化为术语存在概率，并采用课程学习提升检索精度。为解决术语干预任务中数据匮乏问题，我们构建并发布了一个含术语的新语音数据集，以支持后续研究。实验表明，Attention2Probability在测试集上显著优于VectorDB方法，中文与英文术语最大召回率分别达到92.57%和86.83%，每查询延迟仅8.71ms。在SLMs中引入该方法生成的术语可使术语准确率提升6-17%，同时揭示当前SLMs对术语的利用仍存在局限。
原文摘要 · Abstract (English)
Recent advances in speech large language models (SLMs) have improved speech recognition and translation in general domains, but accurately generating domain-specific terms or neologisms remains challenging. To address this, we propose Attention2Probability: attention-driven terminology probability estimation for robust speech-to-text system, which is lightweight, flexible, and accurate. Attention2Probability converts cross-attention weights between speech and terminology into presence probabilities, and it further employs curriculum learning to enhance retrieval accuracy. Furthermore, to tackle the lack of data for speech-to-text tasks with terminology intervention, we create and release a new speech dataset with terminology to support future research in this area. Experimental results show that Attention2Probability significantly outperforms the VectorDB method on our test set. Specifically, its maximum recall rates reach 92.57% for Chinese and 86.83% for English. This high recall is achieved with a latency of only 8.71ms per query. Intervening in SLMs' recognition and translation tasks using Attention2Probability-retrieved terms improves terminology accuracy by 6-17%, while revealing that the current utilization of terminology by SLMs has limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。