让语音大模型实时生成,还能准确识别长语音。
TRADE: Transducer-Augmented Decoder for Speech LLM

- 用声学对齐分支增强语言模型,实现语音帧同步生成。
- 流式识别误差率8.40%,长语音识别误差率3.64%。
- 适合需要低延迟、高精度的语音交互系统使用。
语音大语言模型缺乏可扩展的流式推理机制:其标签同步生成方式无音频帧对齐,导致实时解码与语句结束检测困难。本文提出TRADE(Transducer-Augmented Decoder),通过共享音频编码器并直接使用大语言模型隐藏状态作为预测网络,将帧同步声学对齐与语言推理耦合。三个设计提升性能与实用性:(1) 紧密耦合的双词汇表——从语言模型词汇派生紧凑的声学词汇,实现零成本得分融合;(2) 块同步流式训练配合梯度截断,在不增加内存的情况下消除训练-推理差异;(3) 局部解码器音频注意力(LDAA),采用因果滑动窗口,独立于语句长度限制键值缓存。单一TRADE检查点支持从离线到流式的全范围延迟配置。在Open ASR Leaderboard上平均词错误率(WER)达6.71%,960ms块大小下流式识别率达8.40%。长语音任务中,TED-LIUM上为3.64%,Earnings-22上为10.88%,无需外部分割。同时提供句子结束标点时间戳,结合声学语音活动检测(VAD)后,端到端语句结束检测的F₁提升0.03。
原文摘要 · Abstract (English)
Speech Large Language Models (Speech LLMs) lack a principled mechanism for streaming inference: their label-synchronous generation has no acoustic-frame alignment, making real-time decoding and end-of-utterance detection difficult. We propose TRADE TRansducer-Augmented DEcoder, which augments a multimodal LLM with a transducer branch that shares the audio encoder and uses the LLM's hidden states directly as the prediction network -- coupling frame-synchronous acoustic alignment with the LLM's linguistic reasoning. Three design choices make the system accurate, streamable, and long-form capable: (1)Tightly coupled dual vocabularies -- a compact transducer vocabulary derived from the LLM vocabulary, enabling zero-cost score fusion; (2)Chunk-synchronized streaming training with gradient stopping, eliminating the train-inference mismatch at offline-equivalent memory cost; and (3)Localized Decoder Audio Attention (LDAA), a causal sliding window that caps KV-cache memory independently of utterance length. A single TRADE checkpoint supports offline and streaming decoding across a continuous range of latency operating points. TRADE achieves 6.71% average WER on the Open ASR Leaderboard, while the streaming recognition with 960ms chunk size reaches 8.40% from the same checkpoint. On long-form speech, it obtains 3.64% WER on TED-LIUM and 10.88% on Earnings-22 without external segmentation. TRADE provides sentence-end punctuation timestamps that, when combined with acoustic voice activity detection (VAD), improve end-of-utterance detection by +0.03 F_1 over acoustic VAD alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。