arXiv:2409.19990eess.AScs.CL2024-09被引 5

让语音系统提前预测说话内容并判断何时结束,提升对话响应速度。

Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems

论文配图:Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems
图 1 · 摘自论文原文
  • 用中间段语音预测后续内容,通过掩码未来片段训练模型。
  • 可提前300毫秒预测说话结束时间,且提升识别准确率。
  • 适合需要低延迟响应的智能对话系统开发人员。

有效的语音对话系统应具备自然流畅的交互节奏,类似人类交流。为缩短响应时间,以往研究聚焦于降低自动语音识别(ASR)延迟,但需等待说话人完全说完才能开始处理,限制了自然语言处理(NLP)的时间。人类在听别人说话时会持续预判回应时机。本文首次探索模拟这种预期行为的对话系统,提出利用说话中段预测即将出现的词汇并估计说话结束时间(EOU)。为此,设计一种基于编码器-解码器结构的ASR训练策略:对语音序列的未来部分进行掩码,迫使解码器预测被遮蔽内容;同时引入融合声学与语言信息的交叉注意力算法,精准检测EOU。实验表明,该模型能在实际结束前300毫秒预测出未来事件,并带来整体ASR性能提升。

原文摘要 · Abstract (English)

Effective spoken dialog systems should facilitate natural interactions with quick and rhythmic timing, mirroring human communication patterns. To reduce response times, previous efforts have focused on minimizing the latency in automatic speech recognition (ASR) to optimize system efficiency. However, this approach requires waiting for ASR to complete processing until a speaker has finished speaking, which limits the time available for natural language processing (NLP) to formulate accurate responses. As humans, we continuously anticipate and prepare responses even while the other party is still speaking. This allows us to respond appropriately without missing the optimal time to speak. In this work, as a pioneering study toward a conversational system that simulates such human anticipatory behavior, we aim to realize a function that can predict the forthcoming words and estimate the time remaining until the end of an utterance (EOU), using the middle portion of an utterance. To achieve this, we propose a training strategy for an encoder-decoder-based ASR system, which involves masking future segments of an utterance and prompting the decoder to predict the words in the masked audio. Additionally, we develop a cross-attention-based algorithm that incorporates both acoustic and linguistic information to accurately detect the EOU. The experimental results demonstrate the proposed model's ability to predict upcoming words and estimate future EOU events up to 300ms prior to the actual EOU. Moreover, the proposed training strategy exhibits general improvements in ASR performance.

语音识别对话系统实时预测端到端

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。