一个模型同时生成音乐、补全缺失音轨、按文本提取任意乐器
MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction
- 将分离与补全统一为潜在空间的条件填空任务
- 支持跨数据集联合训练，无需预设乐器类别
- 可按文本指令提取任意乐器，灵活应对复杂场景
我们提出 MGE-LDM，一种统一的潜在扩散框架，可同时实现音乐生成、源信号补全和查询驱动的源分离。与以往受限于固定乐器类别的方法不同，MGE-LDM 在单一紧凑的潜在扩散模型中学习完整混音、子混音及单个音轨的联合分布。推理时，该模型支持（1）完整混音生成，（2）部分生成（即源信号补全），（3）基于文本条件提取任意源。通过将分离与补全均建模为潜在空间中的条件填空任务，本方法实现了对任意乐器源的灵活、无类别依赖操作。值得注意的是，MGE-LDM 可在异构多轨数据集（如 Slakh2100、MUSDB18、MoisesDB）上联合训练，无需依赖预定义的乐器类别。音频样本可在项目主页查看：https://yoongi43.github.io/MGELDM_Samples/。
原文摘要 · Abstract (English)
We present MGE-LDM, a unified latent diffusion framework for simultaneous music generation, source imputation, and query-driven source separation. Unlike prior approaches constrained to fixed instrument classes, MGE-LDM learns a joint distribution over full mixtures, submixtures, and individual stems within a single compact latent diffusion model. At inference, MGE-LDM enables (1) complete mixture generation, (2) partial generation (i.e., source imputation), and (3) text-conditioned extraction of arbitrary sources. By formulating both separation and imputation as conditional inpainting tasks in the latent space, our approach supports flexible, class-agnostic manipulation of arbitrary instrument sources. Notably, MGE-LDM can be trained jointly across heterogeneous multi-track datasets (e.g., Slakh2100, MUSDB18, MoisesDB) without relying on predefined instrument categories. Audio samples are available at our project page: https://yoongi43.github.io/MGELDM_Samples/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。