用文本生成音乐,同时结合局部和全局语义信息提升质量。
Diffusion based Text-to-Music Generation with Global and Local Text based Conditioning
- 通过T5和CLAP双路径提取文本特征,分别对应局部与全局语义。
- 仅用T5的均值池化即可生成高质量音乐,FAD达1.89,优于依赖CLAP的模型。
- 减少模型参数,不依赖额外编码器,适合资源受限场景。
基于扩散模型的文本到音乐生成方法通常依赖预训练大语言模型或跨模态音文表示模型生成文本嵌入。本文提出一种新方法,将UNet同时条件于单模态语言模型(如T5)的局部文本嵌入和跨模态音频-语言模型(如CLAP)的全局嵌入,通过交叉注意力和特征式线性调制(FiLM)实现联合建模。为避免额外编码器,本文还设计了均值池化与自注意力池化机制,从T5中直接提取全局表示。实验表明,加入CLAP全局嵌入可提升文本契合度(KL=1.47),优于仅使用T5的基线模型(KL=1.54)。而仅使用T5均值池化的方案虽文本契合度略低(KL=1.51),但生成质量更优(FAD=1.89),优于联合使用CLAP与T5的模型(FAD=1.94)。该方法在保持性能的同时显著降低参数量。
原文摘要 · Abstract (English)
Diffusion based Text-To-Music (TTM) models generate music corresponding to text descriptions. Typically UNet based diffusion models condition on text embeddings generated from a pre-trained large language model or from a cross-modality audio-language representation model. This work proposes a diffusion based TTM, in which the UNet is conditioned on both (i) a uni-modal language model (e.g., T5) via cross-attention and (ii) a cross-modal audio-language representation model (e.g., CLAP) via Feature-wise Linear Modulation (FiLM). The diffusion model is trained to exploit both a local text representation from the T5 and a global representation from the CLAP. Furthermore, we propose modifications that extract both global and local representations from the T5 through pooling mechanisms that we call mean pooling and self-attention pooling. This approach mitigates the need for an additional encoder (e.g., CLAP) to extract a global representation, thereby reducing the number of model parameters. Our results show that incorporating the CLAP global embeddings to the T5 local embeddings enhances text adherence (KL=1.47) compared to a baseline model solely relying on the T5 local embeddings (KL=1.54). Alternatively, extracting global text embeddings directly from the T5 local embeddings through the proposed mean pooling approach yields superior generation quality (FAD=1.89) while exhibiting marginally inferior text adherence (KL=1.51) against the model conditioned on both CLAP and T5 text embeddings (FAD=1.94 and KL=1.47). Our proposed solution is not only efficient but also compact in terms of the number of parameters required.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。