ACE-Step实现4分钟音乐20秒生成,兼具速度与音乐质量。
ACE-Step: A Step Towards Music Generation Foundation Model
- 融合扩散模型与轻量线性变压器,提升生成效率与结构连贯性。
- 单次生成4分钟音乐仅需20秒(A100 GPU),快15倍于传统方法。
- 支持歌词编辑、人声克隆等精细控制,适合音乐创作者使用。
我们提出ACE-Step,一个开源音乐生成基础模型,通过整体架构设计克服现有方法在生成速度、音乐连贯性与可控性之间的固有权衡。当前方法存在明显局限:基于LLM的模型(如Yue、SongGen)虽能良好对齐歌词,但推理慢且易产生结构瑕疵;扩散模型(如DiffRhythm)虽合成速度快,却常缺乏长程结构连贯性。ACE-Step结合基于扩散的生成、Sana的深度压缩自编码器(DCAE)及轻量线性变压器,并在训练中利用MERT和m-hubert对齐语义表征(REPA),实现快速收敛。结果表明,该模型可在A100 GPU上20秒内生成长达4分钟的音乐,速度比基于LLM的基线快15倍,同时在旋律、和声、节奏等多项指标上实现更优的音乐连贯性与歌词对齐表现。此外,模型保留了细粒度声学细节,支持人声克隆、歌词编辑、混音、轨道生成(如lyric2vocal、singing2accompaniment)等高级控制能力。我们旨在构建一个通用、高效且灵活的音乐AI基础模型,便于在其上训练各类子任务,推动创作工具无缝融入音乐人、制作人与内容创作者的工作流。简言之,目标是打造音乐领域的稳定扩散时刻。代码、模型权重与演示地址:https://ace-step.github.io/
原文摘要 · Abstract (English)
We introduce ACE-Step, a novel open-source foundation model for music generation that overcomes key limitations of existing approaches and achieves state-of-the-art performance through a holistic architectural design. Current methods face inherent trade-offs between generation speed, musical coherence, and controllability. For example, LLM-based models (e.g. Yue, SongGen) excel at lyric alignment but suffer from slow inference and structural artifacts. Diffusion models (e.g. DiffRhythm), on the other hand, enable faster synthesis but often lack long-range structural coherence. ACE-Step bridges this gap by integrating diffusion-based generation with Sana's Deep Compression AutoEncoder (DCAE) and a lightweight linear transformer. It also leverages MERT and m-hubert to align semantic representations (REPA) during training, allowing rapid convergence. As a result, our model synthesizes up to 4 minutes of music in just 20 seconds on an A100 GPU-15x faster than LLM-based baselines-while achieving superior musical coherence and lyric alignment across melody, harmony, and rhythm metrics. Moreover, ACE-Step preserves fine-grained acoustic details, enabling advanced control mechanisms such as voice cloning, lyric editing, remixing, and track generation (e.g. lyric2vocal, singing2accompaniment). Rather than building yet another end-to-end text-to-music pipeline, our vision is to establish a foundation model for music AI: a fast, general-purpose, efficient yet flexible architecture that makes it easy to train subtasks on top of it. This paves the way for the development of powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. In short, our goal is to build a stable diffusion moment for music. The code, the model weights and the demo are available at: https://ace-step.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。