MamTra融合Mamba与Transformer，实现高效语音合成。
MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesis
- 采用交错式结构结合Mamba的线性效率与Transformer的建模能力。
- 仅用2%数据训练，推理显存降低34%且音质无损。
- 适合追求低资源高效率语音合成的研究者与开发者。
尽管基于大语言模型的文本到语音系统取得了卓越音质，但其依赖自回归Transformer导致计算复杂度达二次方，严重限制实际应用。线性时间替代方案如Mamba虽具潜力，却常牺牲表达性合成所需的全局上下文。本文提出MamTra，一种交错式Mamba-Transformer框架，旨在融合Mamba的高效性与Transformer的建模优势。我们还引入新颖的知识迁移策略，将预训练Transformer的知识蒸馏至混合架构中，从而规避从零训练的高昂成本。系统性实验确定了最优混合配置，结果表明，MamTra在不损失语音保真度的前提下，推理显存使用量最多降低34%，甚至仅用原始训练数据的2%即可实现。音频样本可访问：https://mamtratts.github.io。
原文摘要 · Abstract (English)
Despite the remarkable quality of LLM-based text-to-speech systems, their reliance on autoregressive Transformers leads to quadratic computational complexity, which severely limits practical applications. Linear-time alternatives, notably Mamba, offer a potential remedy; however, they often sacrifice the global context essential for expressive synthesis. In this paper, we propose MamTra, an interleaved Mamba-Transformer framework designed to leverage the advantages of Mamba's efficiency and Transformers' modeling capability. We also introduce novel knowledge transfer strategies to distill insights from a pretrained Transformer into our hybrid architecture, thereby bypassing the prohibitive costs of training from scratch. Systematic experiments identify the optimal hybrid configuration, and demonstrate that MamTra reduces inference VRAM usage by up to 34% without compromising speech fidelity - even trained on only 2% of the original training dataset. Audio samples are available at https://mamtratts.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。