提出延迟流建模，实现任意输入组合的实时序列生成。
Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling
- 用预处理对齐流数据，解码器仅需按延迟生成输出
- 在语音识别与语音合成任务中达到顶尖性能与低延迟
- 支持任意长序列，适合实时多模态应用
我们提出延迟流建模（DSM），一种灵活的流式多模态序列到序列学习框架。传统序列生成多为离线模式，需完整接收输入后才开始输出；而流式方法需学习何时推进输入或写入输出。DSM将时间对齐提前至预处理阶段，并引入流间延迟，使解码器仅需基于对齐后的输入流即可进行任意输出序列的流式推理。该方法适用于多种序列生成任务：当文本流延迟时对应自动语音识别（ASR），反向则为文语转换（TTS）。我们在两大核心任务上进行了广泛实验，结果表明DSM在保持低延迟的同时达到领先性能，且能处理任意长序列，甚至可媲美离线基线。代码、示例与演示见https://github.com/kyutai-labs/delayed-streams-modeling。
原文摘要 · Abstract (English)
We introduce Delayed Streams Modeling (DSM), a flexible formulation for streaming, multimodal sequence-to-sequence learning. Sequence-to-sequence generation is often cast in an offline manner, where the model consumes the complete input sequence before generating the first output timestep. Alternatively, streaming sequence-to-sequence rely on learning a policy for choosing when to advance on the input stream, or write to the output stream. DSM instead models already time-aligned streams with a decoder-only language model. By moving the alignment to a pre-processing step,and introducing appropriate delays between streams, DSM provides streaming inference of arbitrary output sequences, from any input combination, making it applicable to many sequence-to-sequence problems. In particular, given text and audio streams, automatic speech recognition (ASR) corresponds to the text stream being delayed, while the opposite gives a text-to-speech (TTS) model. We perform extensive experiments for these two major sequence-to-sequence tasks, showing that DSM provides state-of-the-art performance and latency while supporting arbitrary long sequences, being even competitive with offline baselines. Code, samples and demos are available at https://github.com/kyutai-labs/delayed-streams-modeling
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。