用状态空间模型实现无注意力的高效语音克隆，提升部署稳定性。
MambaVoiceCloning: Efficient and Expressive Text-to-Speech via State-Space Modeling and Diffusion Control
- 全状态空间模型构建条件路径，替代注意力与循环结构。
- 在多个数据集上优于基线模型，参数减少至2100万，吞吐提升1.6倍。
- 适合追求低延迟、高可部署性的实时语音合成场景。
MambaVoiceCloning（MVC）探究了扩散模型语音合成中条件路径是否可在推理时完全由状态空间模型（SSM）构成，从而彻底移除文本、节奏和韵律路径中的注意力机制及显式循环层，同时在受控条件下保持或提升音质。MVC结合门控双向Mamba文本编码器、由轻量对齐教师监督的时序双向Mamba，以及带AdaLN调制的表达性Mamba，实现线性时间复杂度O(T)的条件生成，激活内存有界且支持实际有限前瞻流式处理。与以往在推理时仍为混合架构的Mamba-TTS不同，MVC在固定StyleTTS2声谱扩散-声码器基础上，完全去除了基于注意力的持续时间与风格模块。在LJSpeech/LibriTTS上训练，在VCTK、CSS10（ES/DE/FR）及长文本Gutenberg片段上评估，结果显示其在主观评分（MOS/CMOS）、基频均方根误差（F0 RMSE）、梅尔倒谱失真（MCD）和词错误率（WER）上均取得统计上可靠的提升，同时将编码器参数降至2100万，吞吐提升1.6倍。尽管扩散仍是主要延迟来源，但全SSM条件路径显著改善了内存占用、稳定性和部署可行性。
原文摘要 · Abstract (English)
MambaVoiceCloning (MVC) asks whether the conditioning path of diffusion-based TTS can be made fully SSM-only at inference, removing all attention and explicit RNN-style recurrence layers across text, rhythm, and prosody, while preserving or improving quality under controlled conditions. MVC combines a gated bidirectional Mamba text encoder, a Temporal Bi-Mamba supervised by a lightweight alignment teacher discarded after training, and an Expressive Mamba with AdaLN modulation, yielding linear-time O(T) conditioning with bounded activation memory and practical finite look-ahead streaming. Unlike prior Mamba-TTS systems that remain hybrid at inference, MVC removes attention-based duration and style modules under a fixed StyleTTS2 mel-diffusion-vocoder backbone. Trained on LJSpeech/LibriTTS and evaluated on VCTK, CSS10 (ES/DE/FR), and long-form Gutenberg passages, MVC achieves modest but statistically reliable gains over StyleTTS2, VITS, and Mamba-attention hybrids in MOS/CMOS, F0 RMSE, MCD, and WER, while reducing encoder parameters to 21M and improving throughput by 1.6x. Diffusion remains the dominant latency source, but SSM-only conditioning improves memory footprint, stability, and deployability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。