通过调整U-Net权重提升音频生成的频谱细节，无需训练即可增强音质。
Mel-Refine: A Plug-and-Play Approach to Refine Mel-Spectrogram in Audio Generation
- 在推理时动态调节U-Net中高低频组件权重以增强频谱纹理
- 对Tango2模型提升25%性能指标，显著改善音频清晰度
- 即插即用，兼容所有基于扩散模型的文本转音频架构
文本到音频（TTA）模型能够从文本提示生成多样音频，但主流模型主要依赖梅尔频谱图，仍难以生成内容丰富的音频。梅尔频谱图所需的精细细节和纹理常超出模型能力，导致输出模糊或不连贯。本文研究了U-Net在梅尔频谱生成中的关键作用：跳跃连接和主干网络中的高频成分影响纹理与细节，而主干网络的低频成分对扩散去噪过程至关重要。为此，我们提出「Mel-Refine」——一种无需额外训练或微调的即插即用方法，在推理阶段通过调整不同组件权重来增强梅尔频谱的纹理与细节。实验表明，该方法使最新TTA模型Tango2的性能指标提升25%，验证了其有效性。
原文摘要 · Abstract (English)
Text-to-audio (TTA) model is capable of generating diverse audio from textual prompts. However, most mainstream TTA models, which predominantly rely on Mel-spectrograms, still face challenges in producing audio with rich content. The intricate details and texture required in Mel-spectrograms for such audio often surpass the models' capacity, leading to outputs that are blurred or lack coherence. In this paper, we begin by investigating the critical role of U-Net in Mel-spectrogram generation. Our analysis shows that in U-Net structure, high-frequency components in skip-connections and the backbone influence texture and detail, while low-frequency components in the backbone are critical for the diffusion denoising process. We further propose ``Mel-Refine'', a plug-and-play approach that enhances Mel-spectrogram texture and detail by adjusting different component weights during inference. Our method requires no additional training or fine-tuning and is fully compatible with any diffusion-based TTA architecture. Experimental results show that our approach boosts performance metrics of the latest TTA model Tango2 by 25\%, demonstrating its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。