arXiv:2409.19757eess.AScs.SD2024-09

提出高效长语音识别框架,实现推理时上下文学习。

Efficient Long-Form Speech Recognition for General Speech In-Context Learning

论文配图:Efficient Long-Form Speech Recognition for General Speech In-Context Learning
图 1 · 摘自论文原文
  • 用跨注意力与文档自注意力融合上下文信息
  • 相比基线模型字错误率降低8.64%,内存开销更小
  • 无需微调即可实现说话人适配与上下文偏置

我们提出一种新型端到端语音识别方法,实现高效语音在上下文学习(SICL),适用于长语音解码、测试时说话人适配和测试时上下文偏置。具体地,引入具备SICL能力的注意力编码器-解码器模型(SICL-AED),解码器通过语句级交叉注意力高效整合编码器输出,并利用文档级自注意力学习上下文信息。在TEDLIUM3基准数据集上,SICL-AED利用先前解码输出作为上下文示例,相比基线语句级AED模型相对字错误率降低8.64%。同时,在显著降低运行时与内存复杂度的前提下,性能接近传统长序列AED系统。此外,我们提出一种推理时上下文微调(ICFT)技术,进一步提升SICL效果。说话人适配与上下文偏置实验表明,该系统具备通用语音在上下文学习能力:无需特定微调,其说话人适配性能可媲美监督型基线,上下文偏置任务中实体召回率提升64%。

原文摘要 · Abstract (English)

We propose a novel approach to end-to-end automatic speech recognition (ASR) to achieve efficient speech in-context learning (SICL) for (i) long-form speech decoding, (ii) test-time speaker adaptation, and (iii) test-time contextual biasing. Specifically, we introduce an attention-based encoder-decoder (AED) model with SICL capability (referred to as SICL-AED), where the decoder utilizes an utterance-level cross-attention to integrate information from the encoder's output efficiently, and a document-level self-attention to learn contextual information. Evaluated on the benchmark TEDLIUM3 dataset, SICL-AED achieves an 8.64% relative word error rate (WER) reduction compared to a baseline utterance-level AED model by leveraging previously decoded outputs as in-context examples. It also demonstrates comparable performance to conventional long-form AED systems with significantly reduced runtime and memory complexity. Additionally, we introduce an in-context fine-tuning (ICFT) technique that further enhances SICL effectiveness during inference. Experiments on speaker adaptation and contextual biasing highlight the general speech in-context learning capabilities of our system, achieving effective results with provided contexts. Without specific fine-tuning, SICL-AED matches the performance of supervised AED baselines for speaker adaptation and improves entity recall by 64% for contextual biasing task.

语音识别上下文学习长语音推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。