FlashAudio用直线流加速文本转音频,一步生成质量超百步扩散模型。
FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generation
- 用修正流学习直线轨迹,替代传统弯曲采样路径。
- 单步生成音质超越数百步扩散模型,速度达实时400倍。
- 适合追求高速高保真音频生成的开发者和应用落地场景。
近年来,潜在扩散模型(LDMs)显著提升了文本到音频的生成能力,但其迭代采样过程计算开销大,限制了实际部署。尽管基于一致性的蒸馏方法试图实现少步或单步推理,但其单步性能受限于曲线轨迹,无法超越传统扩散模型。本文提出FlashAudio,通过修正流学习直线轨迹以实现快速模拟。为缓解时间步分配效率低和噪声分布不佳的问题,FlashAudio优化了修正流的时间分布,采用双焦点采样器,并提出不可混流,通过分配最小化批量内数据-噪声对的总距离。此外,针对分类器自由引导(CFG)带来的误差累积放大问题,提出锚定优化,通过锚定参考轨迹来精炼引导尺度。在文本到音频生成任务上的实验表明,FlashAudio的单步生成性能超越需数百步采样的扩散模型,在音质上表现更优,且在单张NVIDIA 4090Ti GPU上实现400倍于实时速度的采样速度。代码将开源于https://github.com/liuhuadai/FlashAudio。
原文摘要 · Abstract (English)
Recent advancements in latent diffusion models (LDMs) have markedly enhanced text-to-audio generation, yet their iterative sampling processes impose substantial computational demands, limiting practical deployment. While recent methods utilizing consistency-based distillation aim to achieve few-step or single-step inference, their one-step performance is constrained by curved trajectories, preventing them from surpassing traditional diffusion models. In this work, we introduce FlashAudio with rectified flows to learn straight flow for fast simulation. To alleviate the inefficient timesteps allocation and suboptimal distribution of noise, FlashAudio optimizes the time distribution of rectified flow with Bifocal Samplers and proposes immiscible flow to minimize the total distance of data-noise pairs in a batch vias assignment. Furthermore, to address the amplified accumulation error caused by the classifier-free guidance (CFG), we propose Anchored Optimization, which refines the guidance scale by anchoring it to a reference trajectory. Experimental results on text-to-audio generation demonstrate that FlashAudio's one-step generation performance surpasses the diffusion-based models with hundreds of sampling steps on audio quality and enables a sampling speed of 400x faster than real-time on a single NVIDIA 4090Ti GPU. Code will be available at https://github.com/liuhuadai/FlashAudio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。