一个统一框架,用单一模型搞定语音修复、分离、转换等多任务生成。
QuarkAudio Technical Report
- 基于自回归解码器,用统一离散分词器处理多任务输入。
- 低帧率下实现高质量音频重建,提升生成效率与性能。
- 支持自然语言指令的自由音频编辑,适合多场景应用。
现有音频处理与生成模型多依赖特定架构,导致开发分散且扩展性差。为此,本文提出QuarkAudio,一种基于解码器的自回归语言模型框架,可统一处理多种任务。该框架采用统一的离散音频分词器H-Codec,结合自监督学习表征进行分词与重建,并引入动态帧率机制及48kHz采样率扩展。通过将任务特定条件信息作为解码器的条件序列,以自回归方式预测离散音频标记,支持语音修复(SR)、目标说话人提取(TSE)、语音分离(SS)、语音转换(VC)及语言查询音频源分离(LASS)。此外,还拓展至由自然语言指令引导的通用自由格式音频编辑(包括语音语义编辑与音频事件编辑)。实验表明,H-Codec在低帧率下实现高质量音频重建,显著提升下游生成效率与性能;QuarkAudio在多项任务上达到或超越现有单任务或多任务系统水平。
原文摘要 · Abstract (English)
Many existing audio processing and generation models rely on task-specific architectures, resulting in fragmented development efforts and limited extensibility. It is therefore promising to design a unified framework capable of handling multiple tasks, while providing robust instruction and audio understanding and high-quality audio generation. This requires a compatible paradigm design, a powerful backbone, and a high-fidelity audio reconstruction module. To meet these requirements, this technical report introduces QuarkAudio, a decoder-only autoregressive (AR) LM-based generative framework that unifies multiple tasks. The framework includes a unified discrete audio tokenizer, H-Codec, which incorporates self-supervised learning (SSL) representations into the tokenization and reconstruction process. We further propose several improvements to H-Codec, such as a dynamic frame-rate mechanism and extending the audio sampling rate to 48 kHz. QuarkAudio unifies tasks by using task-specific conditional information as the conditioning sequence of the decoder-only LM, and predicting discrete target audio tokens in an AR manner. The framework supports a wide range of audio processing and generation tasks, including speech restoration (SR), target speaker extraction (TSE), speech separation (SS), voice conversion (VC), and language-queried audio source separation (LASS). In addition, we extend downstream tasks to universal free-form audio editing guided by natural language instructions (including speech semantic editing and audio event editing). Experimental results show that H-Codec achieves high-quality audio reconstruction with a low frame rate, improving both the efficiency and performance of downstream audio generation, and that QuarkAudio delivers competitive or comparable performance to state-of-the-art task-specific or multi-task systems across multiple tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。