用Mamba模型实现语音对话实时双向流式处理
DuplexMamba: Enhancing Real-time Speech Conversations with Duplex and Streaming Capabilities
- 基于Mamba构建端到端语音-文本对话模型，支持输入输出同步处理
- 在语音识别和语音助手评测中性能媲美最新Transformer模型
- 适合需要低延迟实时交互的语音助手、会议系统等场景
实时语音对话对自然高效的人机交互至关重要，需具备双向通信与流式处理能力。传统基于Transformer的对话机器人采用轮次式交互，计算复杂度随输入长度呈二次增长。本文提出DuplexMamba，一种基于Mamba的端到端多模态双向语音对话模型。该模型支持输入与输出同时处理，动态适应实时流式需求。具体而言，我们设计了基于Mamba的语音编码器，并将其与基于Mamba的语言模型结合。此外，引入新颖的双向解码策略，使模型能同步处理输入并生成输出。实验表明，DuplexMamba成功实现了双向与流式功能，在自动语音识别（ASR）任务及语音助手基准测试中性能可与近期多个Transformer模型相当。代码与模型已开源。
原文摘要 · Abstract (English)
Real-time speech conversation is essential for natural and efficient human-machine interactions, requiring duplex and streaming capabilities. Traditional Transformer-based conversational chatbots operate in a turn-based manner and exhibit quadratic computational complexity that grows as the input size increases. In this paper, we propose DuplexMamba, a Mamba-based end-to-end multimodal duplex model for speech-to-text conversation. DuplexMamba enables simultaneous input processing and output generation, dynamically adjusting to support real-time streaming. Specifically, we develop a Mamba-based speech encoder and adapt it with a Mamba-based language model. Furthermore, we introduce a novel duplex decoding strategy that enables DuplexMamba to process input and generate output simultaneously. Experimental results demonstrate that DuplexMamba successfully implements duplex and streaming capabilities while achieving performance comparable to several recently developed Transformer-based models in automatic speech recognition (ASR) tasks and voice assistant benchmark evaluations. Our code and model are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。