融合流匹配与GAN,实现少步高效高保真音频生成
Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generation
- 两阶段框架:先用流匹配训练,再轻量GAN微调加速推理
- 1/2/4步生成即达高保真音质,比现有方法更高效
- 多分辨率网络处理频域系数,提升音频建模能力
现有音频生成主流方法包括生成对抗网络(GAN)和基于扩散的流匹配(Flow Matching)。GAN训练收敛慢,而流匹配需多步推断,计算开销大。本文提出Flow2GAN,一种两阶段框架:先通过改进流匹配学习生成能力,具体包括将目标重构为终点估计以避免空区域速度估计困难,并采用基于谱能的损失缩放强化感知显著的静音区域;在此基础上,引入轻量级GAN微调,实现1/2/4步等少步推理的高质量音频生成。此外,设计多分支网络架构,在不同时间-频率分辨率下处理傅里叶系数,优于以往单分辨率设计。实验表明,该方法从梅尔频谱图或离散音频符号生成高保真音频,相较当前最先进基于GAN与流匹配的方法,在质量-效率权衡上表现更优。在线演示见https://flow2gan.github.io,源码已开源于https://github.com/k2-fsa/Flow2GAN。
原文摘要 · Abstract (English)
Existing dominant methods for audio generation include Generative Adversarial Networks (GANs) and diffusion-based methods like Flow Matching. GANs suffer from slow convergence during training, while diffusion methods require multi-step inference that introduces considerable computational overhead. In this work, we introduce Flow2GAN, a two-stage framework that combines Flow Matching training for learning generative capabilities with GAN fine-tuning for efficient few-step inference. Specifically, given audio's unique properties, we first improve Flow Matching for audio modeling through: 1) reformulating the objective as endpoint estimation, avoiding velocity estimation difficulties when involving empty regions; 2) applying spectral energy-based loss scaling to emphasize perceptually salient quieter regions. Building on these Flow Matching adaptations, we demonstrate that a further stage of lightweight GAN fine-tuning enables us to obtain few-step (e.g., 1/2/4 steps) generators that produce high-quality audio. In addition, we develop a multi-branch network architecture that processes Fourier coefficients at different time-frequency resolutions, which improves the modeling capabilities compared to prior single-resolution designs. Experimental results indicate that our Flow2GAN delivers high-fidelity audio generation from Mel-spectrograms or discrete audio tokens, achieving highly favorable quality-efficiency trade-offs compared to existing state-of-the-art GAN-based and Flow Matching-based methods. Online demo samples are available at https://flow2gan.github.io, and the source code is released at https://github.com/k2-fsa/Flow2GAN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。