用预训练语音模型实现推理时任务组合，提升语音处理质量
SpeechOp: Inference-Time Task Composition for Generative Speech Processing
- 将预训练语音合成模型改造为通用语音处理器
- 推理时组合不同任务，内容保留率达94.3%（优于基线）
- 适合语音增强、跨语言转换等需要高保真的场景
生成式文本转语音（TTS）系统借助海量真实数据取得显著进展，但语音到语音（S2S）任务如语音增强受限于数据，导致生成方法常扭曲语音内容与说话人身份。为此，我们提出SpeechOp，一种多任务潜在扩散模型，可将预训练TTS模型转化为通用语音处理器，在推理时灵活组合多种语音任务。通过适配预训练TTS模型，SpeechOp继承自然语音的丰富理解，加速训练并提升S2S任务质量，同时改善核心TTS性能。我们进一步提出隐式任务组合（ITC）：利用Whisper等ASR生成的转录文本，引导SpeechOp在推理时完成增强任务。ITC通过结合网络规模语音理解与SpeechOp的生成能力，实现了最先进的内容保留效果（94.3%），音频样本见https://justinlovelace.github.io/projects/speechop。
原文摘要 · Abstract (English)
While generative Text-to-Speech (TTS) systems leverage vast ``in-the-wild" data to achieve remarkable success, speech-to-speech processing tasks like enhancement face data limitations, which lead data-hungry generative approaches to distort speech content and speaker identity. To bridge this gap, we present SpeechOp, a multi-task latent diffusion model that transforms pre-trained TTS models into a universal speech processor capable of performing a wide range of speech tasks and composing them in novel ways at inference time. By adapting a pre-trained TTS model, SpeechOp inherits a rich understanding of natural speech, accelerating training and improving S2S task quality, while simultaneously enhancing core TTS performance. Finally, we introduce Implicit Task Composition (ITC), a novel pipeline where ASR-derived transcripts (e.g., from Whisper) guide SpeechOp's enhancement via our principled inference-time task composition. ITC achieves state-of-the-art content preservation by robustly combining web-scale speech understanding with SpeechOp's generative capabilities. Audio samples are available at https://justinlovelace.github.io/projects/speechop
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。