arXiv:2605.27838cs.SD2026-05被引 7

统一生成文本描述的混音音频场景,效果接近真实录音。

Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text

论文配图:Dasheng AudioGen: A Unified Model for Generating Coherent Audio Scenes from Text
图 1 · 摘自论文原文
  • 用多视角描述拆解复杂声景,实现分层精细控制。
  • 高维语义-声学表示让并发音频成分有效分离融合。
  • 通用模型在混音和单类生成上均表现优异,适合多场景应用。

音频生成长期碎片化,语音、音乐和音效由专用模型处理,难以从单一描述联合生成连贯声景。主要瓶颈在于真实混合音频缺乏细粒度监督,且声学表征有限。本文提出 Dasheng AudioGen,一个从文本生成通用混音声景的统一框架。引入结构化多视图字幕,将复杂声景显式分解为互补描述视图,实现对音频层的精细控制。同时采用高维统一语义-声学表示作为共享隐空间,注入语义先验促进跨模态训练收敛,其高维特征空间具备充足容量以有效解耦与融合并发音频成分。在此设计下,简单流匹配DiT即可实现高质量端到端声景生成。我们还建立了全面的评估流程。实验表明,Dasheng AudioGen 在混合音频类别中性能接近真实录音,同时在单类型生成任务中仍保持与专用模型相当的竞争力。演示见 https://nieeim.github.io/Dasheng-AudioGen-Web/。

原文摘要 · Abstract (English)

Audio generation has long been fragmented, with speech, music, and sound effects produced by domain-specific models that fail to jointly generate coherent audio scenes from a single description. The key obstacles are insufficient fine-grained supervision for real-world mixed audio and limited acoustic representations for modeling concurrent audio components. We present Dasheng AudioGen, a unified framework for generating general mixed-audio scenes from text. Dasheng AudioGen introduces structured multi-view captions, which explicitly decouple complex acoustic scenes into complementary description views, thereby enabling fine-grained control over audio layers. Furthermore, we employ a high-dimensional unified semantic-acoustic representation as the shared latent space. It injects semantic priors that facilitate cross-modal training convergence, while its high-dimensional feature space provides sufficient capacity to disentangle and fuse concurrent audio components effectively. With these designs, a simple flow-matching DiT achieves high-quality end-to-end audio scene generation. We also establish a comprehensive evaluation pipeline for audio scene generation. Experiments demonstrate that Dasheng AudioGen achieves performance approaching real-world recordings in mixed-audio categories, while remaining competitive with specialized models in single-type generation tasks. Demos are available at https://nieeim.github.io/Dasheng-AudioGen-Web/.

音频生成多模态统一模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。