让扩散模型在消费级设备上实时生成音乐,还支持现场互动创作。
Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators

- 用块级键值缓存优化扩散过程,提升推理效率。
- 可在普通游戏本上实时运行,延迟低于200ms。
- 适合音乐人、创作者做现场即兴合作,无需复杂硬件。
交互式流媒体音乐生成有望实现离线模型无法做到的现场表演与共创。然而,当前最先进模型仍处于离散自回归(discrete-AR)范式,训练与推理需工业级算力。本文探讨能否将开源广泛支持但非流式的音频扩散模型高效改造为可交互模型,以在消费级硬件上运行。通过分析现代分块外推扩散流水线,我们发现其推理存在关键效率瓶颈,导致计算效率劣于离散自回归模型。为此提出实时音乐扩散模型(LMDMs),通过简单修改生成过程并引入分块键值缓存,使推理复杂度不仅恢复至离散实时模型(LMMs)水平,且进一步超越。相比LMMs,LMDMs还通过新型ARC-forcing机制实现稳定后训练对齐,避免误差累积,无需显式强化学习或奖励模型。我们在文本条件生成、草图音乐合成及即兴合奏等多个创意场景中验证了其应用。最终展示如何将LMDMs作为生成乐器,在真实艺术家-AI协作中充当“生成延迟”工具,本地运行于消费级游戏笔记本,实时转换演奏者即兴旋律并产生可变音色效果。实验表明,系统平均延迟低于200ms,具备实用交互性。
原文摘要 · Abstract (English)
Interactive streaming music generation promises the use of generative models for live performance and co-creation that is impossible with offline models. However, SOTA models exist in the discrete-AR regime, requiring industrial levels of compute for both training and inference. In this work, we investigate whether audio diffusion models, with their wide support in the open-source community but non-streaming bidirectional nature, can be repurposed efficiently into interactive models accessible on consumer hardware. By taking a critical look at the modern pipeline for block-wise outpainting diffusion, we identify critical inefficiencies during inference that result in strictly worse computational efficiency than their discrete-AR counterparts. We propose Live Music Diffusion Models (LMDMs), a simple modification of the generative diffusion process that recovers, and then outperforms, the inference complexity of the discrete Live Music Models (LMMs) through block-wise KV Caching. Unlike LMMs, LMDMs further enable stable post-training alignment through our novel ARC-Forcing paradigm, reducing error accumulation without any explicit RL or reward models. We demonstrate the application of LMDMs in a number of creative domains, including text-conditioned generation, sketch-based music synthesis, and jamming. We finally show how LMDMs can be used as a generative instrument in a real artist-AI collaboration, utilizing LMDMs as a "generative delay" to transform musicians' improvisation live for variable timbral effects while running locally on a consumer gaming laptop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。