用大模型与超分技术生成8分钟高保真音乐，支持文本和音频提示。
InspireMusic: Integrating Super Resolution and Large Language Model for High-Fidelity Long-Form Music Generation
- 融合自回归大模型与超分辨率流匹配，实现长时序音乐生成。
- 支持8分钟高保真音频生成，采样率更高且细节更丰富。
- 适合音乐创作、跨模态生成场景，代码已开源。
我们提出InspireMusic，一个集成超分辨率与大语言模型的高保真长时序音乐生成框架。该框架采用统一架构，结合自回归Transformer与超分辨率流匹配模型，可基于文本或音频提示，可控生成高保真音乐、歌曲及音频。不同于以往方法，我们使用单一码本音频分词器，包含更丰富的语义信息，降低训练成本并提升效率。该设计使模型在8分钟长序列上仍保持高质量连贯性。基于Qwen 2.5的自回归变压器模型用于预测音频标记，随后通过超分辨率流匹配模型，利用声学编码器学习的精细细节生成高采样率音频。综合实验表明，InspireMusic-1.5B-Long模型在主观与客观评估中表现媲美MusicGen与Stable Audio 2.0等顶尖开源系统。代码与预训练模型已发布于https://github.com/FunAudioLLM/InspireMusic。
原文摘要 · Abstract (English)
We introduce InspireMusic, a framework integrated super resolution and large language model for high-fidelity long-form music generation. A unified framework generates high-fidelity music, songs, and audio, which incorporates an autoregressive transformer with a super-resolution flow-matching model. This framework enables the controllable generation of high-fidelity long-form music at a higher sampling rate from both text and audio prompts. Our model differs from previous approaches, as we utilize an audio tokenizer with one codebook that contains richer semantic information, thereby reducing training costs and enhancing efficiency. This combination enables us to achieve high-quality audio generation with long-form coherence of up to $8$ minutes. Then, an autoregressive transformer model based on Qwen 2.5 predicts audio tokens. Next, we employ a super-resolution flow-matching model to generate high-sampling rate audio with fine-grained details learned from an acoustic codec model. Comprehensive experiments show that the InspireMusic-1.5B-Long model has a comparable performance to recent top-tier open-source systems, including MusicGen and Stable Audio 2.0, on subjective and objective evaluations. The code and pre-trained models are released at https://github.com/FunAudioLLM/InspireMusic.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。