arXiv:2509.24391cs.SD2025-09被引 15

统一音频生成框架,支持文本/语音/视频多模态输入。

UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities

  • 基于流匹配构建统一框架,融合时序对齐与非对齐任务。
  • 7个任务表现优异,训练数据少于8000小时,参数低于10亿。
  • 非自回归设计,小模型(200M)也具竞争力,适合基础模型应用。

音频生成近年来发展迅速,涵盖语音、音乐和音效等任务。这些任务可分为时序对齐(TA)和非时序对齐(NTA)两类,传统方法因建模范式不同而分道扬镳。但音频本质无此划分,统一建模成为通用音频生成的必然方向。此前统一方法多采用自回归架构,非自回归路径仍待探索。本文提出UniFlow-Audio,基于流匹配的统一音频生成框架,设计双融合机制:在每层中通过时间对齐对齐音频隐变量与TA特征,并用交叉注意力整合NTA特征。采用任务平衡采样策略,确保两类任务性能均衡。该框架支持文本、音频、视频等多模态输入。得益于多任务学习与流匹配的生成能力,仅需少于8000小时公开数据和不足10亿可训练参数,便在7个任务上取得强效果。即使是仅约200M参数的小型版本,性能亦具竞争力,展现出作为非自回归音频基础模型的潜力。代码与模型将发布于https://wsntxxn.github.io/uniflow_audio。

原文摘要 · Abstract (English)

Audio generation, including speech, music and sound effects, has advanced rapidly in recent years. These tasks can be divided into two categories: time-aligned (TA) tasks, where each input unit corresponds to a specific segment of the output audio (e.g., phonemes aligned with frames in speech synthesis); and non-time-aligned (NTA) tasks, where such alignment is not available. Since modeling paradigms for the two types are typically different, research on different audio generation tasks has traditionally followed separate trajectories. However, audio is not inherently divided into such categories, making a unified model a natural and necessary goal for general audio generation. Previous unified audio generation works have adopted autoregressive architectures, while unified non-autoregressive approaches remain largely unexplored. In this work, we propose UniFlow-Audio, a universal audio generation framework based on flow matching. We propose a dual-fusion mechanism that temporally aligns audio latents with TA features and integrates NTA features via cross-attention in each model block. Task-balanced data sampling is employed to maintain strong performance across both TA and NTA tasks. UniFlow-Audio supports omni-modalities, including text, audio, and video. By leveraging the advantage of multi-task learning and the generative modeling capabilities of flow matching, UniFlow-Audio achieves strong results across 7 tasks using fewer than 8K hours of public training data and under 1B trainable parameters. Even the small variant with only ~200M trainable parameters shows competitive performance, highlighting UniFlow-Audio as a potential non-auto-regressive foundation model for audio generation. Code and models will be available at https://wsntxxn.github.io/uniflow_audio.

音频生成流匹配多模态非自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。