用轻量机制实现音乐生成的精准时序控制，参数减少6.75倍。
MuseControlLite: Multifunctional Music Generation with Lightweight Conditioners
- 在解耦交叉注意力中加入旋转位置编码，提升时间依赖条件控制
- 仅用8500万参数，控制准确率达61.1%，较顶尖方法少6.75倍参数
- 支持旋律、音频修复等多任务控制，适合资源有限的音乐生成应用
我们提出MuseControlLite，一种轻量级机制，用于微调文本到音乐生成模型，实现对多种时变音乐属性和参考音频信号的精确控制。关键发现是：尽管位置编码在文本到音乐生成模型的条件器中很少被使用，但在条件依赖于时间时却至关重要。以旋律控制为例，实验表明，仅在解耦的交叉注意力层中添加旋转位置编码，即可将控制准确率从56.6%提升至61.1%，同时所需可训练参数仅为当前最先进微调方法的1/6.75，且使用相同的Stable Audio Open预训练扩散Transformer模型。我们评估了多种音乐属性控制、音频修复与音频外推任务，结果表明在显著更低的微调成本下，其可控性优于MusicGen-Large和Stable Audio Open ControlNet，仅需8500万可训练参数。源代码、模型检查点及演示示例详见：https://musecontrollite.github.io/web/
原文摘要 · Abstract (English)
We propose MuseControlLite, a lightweight mechanism designed to fine-tune text-to-music generation models for precise conditioning using various time-varying musical attributes and reference audio signals. The key finding is that positional embeddings, which have been seldom used by text-to-music generation models in the conditioner for text conditions, are critical when the condition of interest is a function of time. Using melody control as an example, our experiments show that simply adding rotary positional embeddings to the decoupled cross-attention layers increases control accuracy from 56.6% to 61.1%, while requiring 6.75 times fewer trainable parameters than state-of-the-art fine-tuning mechanisms, using the same pre-trained diffusion Transformer model of Stable Audio Open. We evaluate various forms of musical attribute control, audio inpainting, and audio outpainting, demonstrating improved controllability over MusicGen-Large and Stable Audio Open ControlNet at a significantly lower fine-tuning cost, with only 85M trainble parameters. Source code, model checkpoints, and demo examples are available at: https://musecontrollite.github.io/web/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。