让语音建模实时运行，语音和文字长度对齐且低延迟。
TASTE-Streaming: Towards Streamable Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling
- 用CTC模块实现语音文字同步编码，无需依赖外部ASR。
- 设计可流式解码的单元解码器，延迟显著降低。
- 适合实时语音交互系统，尤其长语音场景表现稳定。
文本-语音联合语音语言建模旨在实现自然智能的语音交互，但语音单元序列通常远长于文本标记，造成模态不匹配。现有方法通过文本对齐的语音分词与嵌入（TASTE）缓解该问题，但依赖外部ASR且使用非因果解码器，难以支持流式处理。为此，我们提出TASTE-S，一种适用于实时场景的可流式扩展版本。TASTE-S在编码器中集成基于CTC的ASR模块，实现即时双模态编码；并重新设计单元解码器以支持在线解码。联合训练表明，TASTE-S性能媲美TASTE，同时大幅降低延迟。进一步实验显示，TASTE-S对转录错误具有鲁棒性，支持长序列编码与解码。
原文摘要 · Abstract (English)
Text-speech joint spoken language modeling (SLM) aims at natural and intelligent speech-based interactions, but developing such a system may suffer from modality mismatch: speech unit sequences are much longer than text tokens. Prior work reduces this gap with text-aligned tokenization and embedding (TASTE), producing speech tokens that align in lengths with their textual counterparts. However, the dependence on an external ASR system and the use of a non-causal decoder limits streaming use. To address this limitation, we propose TASTE-S, a streamable extension of TASTE suitable for real-time usage. TASTE-S integrates a CTC-based ASR module into the encoder for instant dual-modality encoding. We also redesign the unit decoder to enable on-the-fly decoding. With joint training, we show that TASTE-S matches TASTE's performance while significantly reducing latency. Further investigations reveal that TASTE-S remains robust to transcriptions and enables long-form encoding and decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。