arXiv:2409.00587cs.SDcs.CV2024-09被引 26

用优化的流模型实现更自然的文生音乐,效果超越传统扩散模型。

FLUX that Plays Music

  • 基于修正流的Transformer架构,分步处理文本与音乐序列。
  • 在自动评分和人工评测中均显著优于现有扩散模型。
  • 适合对音乐生成、文本条件建模感兴趣的开发者与研究者。

本文提出一种基于扩散修正流的文本到音乐生成方法FluxMusic。通过将先进的Flux模型迁移至梅尔频谱的潜在变分自编码器空间,先对双流文本-音乐序列独立应用多头注意力,再堆叠单一音乐流进行去噪补丁预测。采用多个预训练文本编码器充分捕捉描述语义信息,并结合时间步嵌入,通过调制机制引入粗粒度文本信息,细粒度文本细节则直接拼接至音乐补丁序列作为输入。实验表明,经过优化架构的修正流训练在自动指标和人类偏好评估中均显著优于现有扩散方法。相关代码与模型权重已公开于:https://github.com/feizc/FluxMusic。

原文摘要 · Abstract (English)

This paper explores a simple extension of diffusion-based rectified flow Transformers for text-to-music generation, termed as FluxMusic. Generally, along with design in advanced Flux\footnote{https://github.com/black-forest-labs/flux} model, we transfers it into a latent VAE space of mel-spectrum. It involves first applying a sequence of independent attention to the double text-music stream, followed by a stacked single music stream for denoised patch prediction. We employ multiple pre-trained text encoders to sufficiently capture caption semantic information as well as inference flexibility. In between, coarse textual information, in conjunction with time step embeddings, is utilized in a modulation mechanism, while fine-grained textual details are concatenated with the music patch sequence as inputs. Through an in-depth study, we demonstrate that rectified flow training with an optimized architecture significantly outperforms established diffusion methods for the text-to-music task, as evidenced by various automatic metrics and human preference evaluations. Our experimental data, code, and model weights are made publicly available at: \url{https://github.com/feizc/FluxMusic}.

文生音乐修正流音频生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。