开源音乐生成模型,可在消费级显卡上快速生成高质量歌曲。
ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation
- 用语言模型做全能创作规划,生成从短循环到10分钟完整曲子的蓝图。
- 单首歌生成时间低于2秒(A100)或10秒(RTX 3090),VRAM占用少于4GB。
- 支持仅用几首歌微调风格,可实现翻唱、配乐替换等精细编辑功能。
我们提出ACE-Step v1.5,一个高效开源的音乐基础模型,将商业级生成能力带至消费级硬件。在常用评估指标下,其生成质量超越多数商用音乐模型,同时保持极高速度——在A100上单首歌生成时间低于2秒,在RTX 3090上低于10秒。模型本地运行所需显存不足4GB,支持轻量级个性化:用户仅需几首歌曲即可训练LoRA以捕捉自身风格。核心采用新型混合架构,语言模型(LM)作为全功能规划器,将简单用户指令转化为完整歌曲蓝图,涵盖结构、元数据、歌词与描述,并通过思维链引导扩散变压器(DiT)。该对齐机制基于内在强化学习,仅依赖模型内部机制,避免外部奖励模型或人工偏好带来的偏差。除标准合成外,模型还统一实现精确风格控制与多样编辑功能,如翻唱生成、配色重绘及人声转伴奏,严格遵循50多种语言的提示。代码、模型权重与演示已公开。
原文摘要 · Abstract (English)
We present ACE-Step v1.5, a highly efficient open-source music foundation model that brings commercial-grade generation to consumer hardware. On commonly used evaluation metrics, ACE-Step v1.5 achieves quality beyond most commercial music models while remaining extremely fast -- under 2 seconds per full song on an A100 and under 10 seconds on an RTX 3090. The model runs locally with less than 4GB of VRAM, and supports lightweight personalization: users can train a LoRA from just a few songs to capture their own style. At its core lies a novel hybrid architecture where the Language Model (LM) functions as an omni-capable planner: it transforms simple user queries into comprehensive song blueprints -- scaling from short loops to 10-minute compositions -- while synthesizing metadata, lyrics, and captions via Chain-of-Thought to guide the Diffusion Transformer (DiT). Uniquely, this alignment is achieved through intrinsic reinforcement learning relying solely on the model's internal mechanisms, thereby eliminating the biases inherent in external reward models or human preferences. Beyond standard synthesis, ACE-Step v1.5 unifies precise stylistic control with versatile editing capabilities -- such as cover generation, repainting, and vocal-to-BGM conversion -- while maintaining strict adherence to prompts across 50+ languages. This paves the way for powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. The code, the model weights and the demo are available at: https://ace-step.github.io/ace-step-v1.5.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。