用块生成+深度解码,实现超低延迟端到端语音合成。
An Ultra-Low Latency, End-to-End Streaming Speech Synthesis Architecture via Block-Wise Generation and Depth-Wise Codec Decoding
- 直接建模Mimi压缩离散潜空间,跳过传统声码器瓶颈。
- 首字节延迟仅48.99毫秒,比传统方案快10.6倍。
- 适合实时交互场景,如语音助手、直播语音等。
实时语音合成需在推理延迟与音质间取得平衡。传统连续文本转语音流程依赖计算量大的神经声码器重建相位信息,形成显著流式瓶颈。此外,回归式声学建模常导致频谱过度平滑。为此,本文提出一种面向超低延迟的端到端非自回归架构,通过块式生成与深度解码策略,直接建模Mimi神经音频编解码器的高压缩离散潜空间。结合改进的FastSpeech 2骨干网络与渐进式深度序列解码机制,动态条件化32层残差向量量化码本。该方法有效缓解音素对齐退化,无需时序自回归开销即可管理高保真离散表示复杂度。在英语和马来语数据集上的实验验证了其语言无关部署能力。相比传统连续回归模型,该架构在基频发声准确率上实现量化提升,并减轻高频频谱失真。系统实现超低延迟推理,相较传统级联流水线绝对加速10.6倍,平均首字节延迟为48.99毫秒,远低于人类感知阈值,确立其作为实时流式语音接口的高效解决方案。
原文摘要 · Abstract (English)
Real-time speech synthesis requires balancing inference latency and acoustic fidelity for interactive applications. Conventional continuous text-to-speech pipelines require computationally intensive neural vocoders to reconstruct phase information, creating a significant streaming bottleneck. Furthermore, regression-based acoustic modeling frequently induces spectral over-smoothing artifacts. To address these limitations, this paper proposes a novel end-to-end non-autoregressive architecture optimized for ultra-low latency block-wise generation, directly modeling the highly compressed discrete latent space of the Mimi neural audio codec. Integrating a modified FastSpeech 2 backbone with a progressive depth-wise sequential decoding strategy, the architecture dynamically conditions 32 layers of residual vector quantization codes. This mechanism resolves phonetic alignment degradation and manages the complexity of high-fidelity discrete representations without temporal autoregressive overhead. Experimental evaluations on English and Malay datasets validate its language-independent deployment capability. Compared to conventional continuous regression models, the proposed architecture demonstrates quantitative improvements in fundamental voicing accuracy and mitigates high-frequency spectral degradation. It achieves ultra-low latency inference, translating to a 10.6-fold absolute acceleration over conventional cascaded pipelines. Crucially, the system achieves an average time-to-first-byte latency of 48.99 milliseconds, falling significantly below the human perception threshold for real-time interactive streaming. These results firmly establish the proposed architecture as a highly optimized solution for deploying real-time streaming speech interfaces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。