用统一框架让语言模型搞定语音增强各类任务
UniSE: A Unified Framework for Decoder-Only Autoregressive LM-Based Speech Enhancement
- 基于解码器的自回归语言模型，统一处理语音修复、提取和分离
- 在多个基准上表现优于传统方法，语音质量提升明显
- 适合想用统一模型解决多类语音增强问题的研究者
神经音频编解码器推动了语言模型在语音应用中的使用。然而，自回归语言模型在统一语音增强（SE）任务方面的潜力尚未充分探索。本文提出 UniSE，一种基于解码器的统一语言模型框架，可处理语音修复、目标说话人提取和语音分离等不同任务。该模型以输入语音特征为条件，自回归生成目标离散标记，实现多种任务学习模式的兼容。为进一步优化语音质量，引入基于多评估标准的渐进式强化学习策略。在多个基准上的实验表明，UniSE 的性能与判别式和生成式基线相当，证明了语言模型在统一语音增强任务中的能力。代码与演示见：https://github.com/alibaba/unified-audio/tree/main/QuarkAudio-UniSE。
原文摘要 · Abstract (English)
Neural audio codecs have largely promoted the application of language models (LMs) for speech applications. However, the effectiveness of autoregressive LM-based models in unifying speech enhancement (SE) tasks remains underexplored. In this work, we propose UniSE, a unified decoder-only LM-based framework to handle different SE tasks including speech restoration, target speaker extraction, and speech separation. Conditioned on input speech features, it autoregressively generates target discrete tokens, facilitating compatibility between distinct learning patterns of multiple tasks. To further optimize speech quality, we introduce a progressive reinforcement learning strategy with multiple assessment criteria. Experiments on several benchmarks show that UniSE achieves competitive performance compared to discriminative and generative baselines, demonstrating the capacity of LMs in unifying SE tasks. The code and demo are available at: https://github.com/alibaba/unified-audio/tree/main/QuarkAudio-UniSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。