用跨注意力Mamba实现高保真语音编辑与零样本语音合成
Speak, Edit, Repeat: High-Fidelity Voice Editing and Zero-Shot TTS with Cross-Attentive Mamba
- 基于跨注意力Mamba架构，实现高效语音建模与文本-语音精准对齐
- 语音编辑在40样本测试中57.2%被认作与原声无差别，优于现有模型
- 内存占用仅为VoiceCraft的1/6，适合实时部署且无需后处理
我们提出MAVE（带交叉注意力的Mamba语音编辑与合成模型），一种基于交叉注意力Mamba主干的自回归架构，用于文本条件语音编辑和高保真语音合成。MAVE在真实世界音频上的语音编辑任务达到顶尖性能，且在零样本语音合成上表现优异，未显式训练于此任务却超越主流自回归与扩散模型。通过结合Mamba的高效序列建模与交叉注意力的精准对齐，MAVE实现上下文感知的自然语音编辑，保持极佳的说话人一致性和流畅度。在RealEdit基准的随机40样本子集上（共400次判断），57.2%的听者认为编辑后语音与原始语音难以区分，24.8%偏好原始语音，18.0%偏好MAVE，表明多数情况下编辑结果不可察觉。相较于VoiceCraft和FluentSpeech，MAVE在配对评估和独立平均意见分（MOS）中均表现更优。零样本语音合成方面，MAVE在说话人相似度和自然度上均超过VoiceCraft，无需多次推理或后处理。令人瞩目的是，其质量提升同时带来显著更低的内存开销：在A100、FP16、批量大小1条件下，对平均时长6.21秒的语句推理，内存仅需VoiceCraft的约1/6，延迟相近。
原文摘要 · Abstract (English)
We introduce MAVE (Mamba with Cross-Attention for Voice Editing and Synthesis), a novel autoregressive architecture for text-conditioned voice editing and high-fidelity text-to-speech (TTS) synthesis, built on a cross-attentive Mamba backbone. MAVE achieves state-of-the-art performance in speech editing and very competitive results in zero-shot TTS, while not being explicitly trained on the latter task, outperforming leading autoregressive and diffusion models on diverse, real-world audio. By integrating Mamba for efficient audio sequence modeling with cross-attention for precise text-acoustic alignment, MAVE enables context-aware voice editing with exceptional naturalness and speaker consistency. In pairwise human evaluations on a random 40-sample subset of the RealEdit benchmark (400 judgments), 57.2% of listeners rated MAVE - edited speech as perceptually equal to the original, while 24.8% prefered the original and 18.0% MAVE - demonstrating that in the majority of cases edits are indistinguishable from the source. MAVE compares favorably with VoiceCraft and FluentSpeech both on pairwise comparisons and standalone mean opinion score (MOS) evaluations. For zero-shot TTS, MAVE exceeds VoiceCraft in both speaker similarity and naturalness, without requiring multiple inference runs or post-processing. Remarkably, these quality gains come with a significantly lower memory cost and approximately the same latency: MAVE requires ~6x less memory than VoiceCraft during inference on utterances from the RealEdit database (mean duration: 6.21s, A100, FP16, batch size 1). Our results demonstrate that MAVE establishes a new standard for flexible, high-fidelity voice editing and synthesis through the synergistic integration of structured state-space modeling and cross-modal attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。