首个统一音频生成、编辑与字幕的扩散模型框架
UAT: Unified Audio-Text Diffusion for Audio Generation, Editing, and Captioning
- 用连续扩散建模音频，离散扩散建模文本，共享双流主干
- 在保持强音频生成能力的同时实现媲美主流的字幕效果
- 适合需要多任务音频处理的研究者与开发者
音频生成与音频到文本的理解长期分离：扩散模型主导高质量合成，自回归语言模型则负责字幕与语义预测。现有统一方法通常依赖异构模块或以自回归为中心的建模，阻碍联合优化并限制声学保真度。我们提出UAT，据知是首个以扩散模型为核心的统一框架，支持音频生成、编辑与字幕生成。UAT将音频的连续潜在扩散与文本的掩码离散扩散相结合，在共享双流主干中实现双向音频-文本建模。实验表明，UAT在保持强大音频生成与编辑能力的同时，达到具有竞争力的字幕性能，展示了声学合成与语义预测之间的良好平衡。演示样本可访问 https://UAT-demo.github.io。
原文摘要 · Abstract (English)
Audio generation and audio-to-text understanding remain largely separate, with diffusion models dominating high-fidelity synthesis and autoregressive (AR) language models driving captioning and semantic prediction. Existing unified approaches typically rely on either heterogeneous modules or AR-centric modeling, which can hinder joint optimization and limit acoustic fidelity. We present UAT, to our knowledge, the first diffusion-centric framework that supports unified audio generation, editing, and captioning. UAT couples continuous latent diffusion for audio with masked discrete diffusion for text, enabling bidirectional audio-text modeling within a shared dual-stream backbone. Experiments show that UAT preserves strong audio generation and editing capabilities while achieving competitive captioning performance, demonstrating a favorable balance between acoustic synthesis and semantic prediction. Demo samples are available at https://UAT-demo.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。