用GAN实现秒级高保真文本转音频，效率远超现有模型
AudioGAN: A Compact and Efficient Framework for Real-Time High-Fidelity Text-to-Audio Generation
- 基于GAN设计单次生成框架，突破传统扩散模型速度瓶颈
- 参数量少90%，推理速度提升20倍，1秒内完成高质量音频合成
- 适合需要实时生成的媒体创作、语音交互等场景
文本到音频（TTA）生成能显著降低媒体行业制作成本并提升效率。然而，当前主流的基于扩散模型的TTA方法存在推理速度慢、计算开销高的问题。本文提出AudioGAN，首个成功基于生成对抗网络（GAN）的TTA框架，可实现单次前向传播生成音频，大幅降低模型复杂度与推理时间。为克服GAN训练难题，引入多种对比损失，并提出创新组件：单-双-三注意力（SDT Attention）与时频交叉注意力（TF-CA）。在AudioCaps数据集上的大量实验表明，AudioGAN达到当前最优性能，参数量减少90%，推理速度提升20倍，音频生成时间低于1秒。该结果确立了AudioGAN作为实时TTA的实用且高效解决方案。
原文摘要 · Abstract (English)
Text-to-audio (TTA) generation can significantly benefit the media industry by reducing production costs and enhancing work efficiency. However, most current TTA models (primarily diffusion-based) suffer from slow inference speeds and high computational costs. In this paper, we introduce AudioGAN, the first successful Generative Adversarial Networks (GANs)-based TTA framework that generates audio in a single pass, thereby reducing model complexity and inference time. To overcome the inherent difficulties in training GANs, we integrate multiple ,contrastive losses and propose innovative components Single-Double-Triple (SDT) Attention and Time-Frequency Cross-Attention (TF-CA). Extensive experiments on the AudioCaps dataset demonstrate that AudioGAN achieves state-of-the-art performance while using 90% fewer parameters and running 20 times faster, synthesizing audio in under one second. These results establish AudioGAN as a practical and powerful solution for real-time TTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。