用纯Transformer替换音频解码器，手机端语音合成延迟从42.1毫秒降至4.4毫秒
T-Mimi: A Transformer-based Mimi Decoder for Real-Time On-Phone TTS
- 用纯Transformer替代原解码器中的卷积层，提升移动端推理效率
- 在手机端实现4.4毫秒低延迟，比原有方案快约90%
- 发现靠近波形的最后两层需保持全精度，否则音质显著下降
神经音频编码器为语音合成提供了有前景的声学特征，代表性流式编码器Mimi能为实时文本转语音（TTS）提供高质量声学特征。然而，Mimi解码器采用混合变压器与卷积结构，其去卷积层计算密集，不适用于移动CPU（如XNNPACK），导致边缘设备上存在显著延迟瓶颈。本文提出T-Mimi，一种对Mimi编码器解码器的新改进，借鉴TS3-Codec架构，将卷积组件完全替换为纯变压器解码器。该修改将设备端TTS延迟从42.1毫秒大幅降低至4.4毫秒。此外，通过量化感知训练发现：靠近波形输出的最后两层变压器和最终线性层对量化极为敏感，必须保持全精度以维持音频质量。
原文摘要 · Abstract (English)
Neural audio codecs provide promising acoustic features for speech synthesis, with representative streaming codecs like Mimi providing high-quality acoustic features for real-time Text-to-Speech (TTS) applications. However, Mimi's decoder, which employs a hybrid transformer and convolution architecture, introduces significant latency bottlenecks on edge devices due to the the compute intensive nature of deconvolution layers which are not friendly for mobile-CPUs, such as the most representative framework XNNPACK. This paper introduces T-Mimi, a novel modification of the Mimi codec decoder that replaces its convolutional components with a purely transformer-based decoder, inspired by the TS3-Codec architecture. This change dramatically reduces on-device TTS latency from 42.1ms to just 4.4ms. Furthermore, we conduct quantization aware training and derive a crucial finding: the final two transformer layers and the concluding linear layers of the decoder, which are close to the waveform, are highly sensitive to quantization and must be preserved at full precision to maintain audio quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。