arXiv:2609.08936cs.SDcs.CL2026-09

一个能用自然语言统一生成和编辑语音的开源大模型。

AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing

论文配图:AuK Technical Report: An Open-Source Foundational Model for Speech Generation and Editing
图 1 · 摘自论文原文
  • 用自然语言指令+音频上下文统一控制语音生成与编辑
  • 训练数据达30亿条指令和195万小时音频,支持五类任务
  • 模型压缩后推理速度快4.5倍,适合实际部署应用

我们提出AuK,一个开源的基础语音生成与编辑模型,通过自然语言指令和音频上下文实现统一控制。为支持广泛能力,构建约30.3亿条指令-音频样本及195万小时有效监督数据,覆盖语音生成、内容编辑、增强分离、副语言编辑和声学编辑五大任务。AuK融合多模态大语言模型进行语义条件建模,采用在语音、通用音频和音乐上联合训练的VAE实现声学条件建模,并使用混合修正流Transformer生成,包含双流MMDiT块与统一单流DiT块。训练先进行仅生成的预热,再进入生成-编辑联合预训练,随后采用人类反馈偏好优化和基于奖励的强化学习进行后训练。为降低推理成本,引入一致性初始化与任务路由解耦DMD的模型蒸馏。最终得到的AuK-Flash可在4步内完成推理,无需分类器自由引导,在相同条件下比全模型快4.5倍。实验表明其在零样本和指令控制语音生成以及通用指令引导编辑上表现领先,同时在信号级修复任务中保持竞争力。源代码与模型权重均已公开,以支持可复现性和进一步研究。

原文摘要 · Abstract (English)

We introduce AuK, an open-source foundational model that unifies speech generation and editing through a common interface of natural-language instructions and audio context. To support this broad capability set, we construct approximately 3.03 billion instruction--audio instances and 1.95 million hours of effective supervision across five task families: speech generation, content editing, enhancement and separation, paralinguistic editing, and acoustic editing. AuK combines a multimodal large language model for semantic conditioning, an VAE jointly trained on speech, general audio, and music for acoustic conditioning, and a hybrid rectified-flow Transformer that performs dual-stream MMDiT blocks followed by unified single-stream DiT blocks for generation. Training begins with generation-only warm-up and proceeds to joint generation--editing pre-training. We then apply complementary post-training strategies: human-feedback preference optimization for open-ended editing and reward-based reinforcement learning for speech generation. To reduce inference cost, we further distill the model with consistency initialization and task-routed Decoupled DMD. The resulting AuK-Flash performs 4-step inference without classifier-free guidance and achieves a 4.5 wall-clock speedup over the full model under matched conditions. Experiments demonstrate leading performance on zero-shot and instruction-controlled speech generation and general instruction-guided editing, while remaining competitive on signal-level restoration tasks. We release both the source code and model weights to support reproducibility and further research.

语音生成编辑模型开源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。