用统一声学符号层次实现音乐结构与细节的渐进生成
Shao: Scaling Acoustic Token Language Models Toward High-Fidelity Music Generation

- 构建64层声学符号表示,分两阶段逐层细化生成
- 62步推理完成全曲精细化,支持并行处理提升效率
- 纯声学符号建模即可实现歌词对齐,无需额外语义层
高质量音乐生成常将结构与保真度分离处理:先建模高层结构,再通过扩散或神经解码重建细节。本文提出新思路:在单一深层声学符号层级中逐步建模两者。构建64层残差向量量化(RVQ)声学表示,设计粗粒度到细粒度的两阶段生成框架。主干模型生成全曲粗粒度声学符号,超分辨率模型在同一符号空间内逐层细化。该阶段以全曲尺度并行运行,每层逐层优化,固定62步推理过程。为兼顾歌词对齐与细节重建,引入混合注意力训练:对齐使用因果注意力,层级细化使用全注意力。关键发现是,仅靠纯声学符号语言建模即可实现文本-人声对齐,无需独立语义符号阶段。此外,从已训练主干模型初始化超分辨率模型可显著提升收敛速度与最终质量。结果表明,无需将结构与保真度分离至异构表示空间,两者可在统一声学符号层级中渐进建模,为高质量音乐生成提供更简洁统一路径。
原文摘要 · Abstract (English)
A common design pattern in high-quality music generation is to handle structure and fidelity in different representation spaces: a generator first models high-level structure, followed by diffusion-based or neural decoding stages that reconstruct fine details. In this work, we explore an alternative view: both may be progressively modeled within a single deep acoustic-token hierarchy. To study this, we build a 64-layer residual vector quantization (RVQ) acoustic representation and propose a two-stage coarse-to-fine generation framework. A backbone model first generates coarse acoustic tokens for the full track, and a super-resolution model then completes finer tokens within the same acoustic token space. The super-resolution stage works at full-track scale and refines tokens layer by layer while running in parallel over time, leading to a fixed 62-step inference process. To jointly improve lyric alignment and fine-detail reconstruction, we further introduce hybrid-attention training: the alignment objective uses causal attention, while layer-wise refinement uses full attention. A key finding is that text-vocal alignment can emerge within pure acoustic-token language modeling, without requiring a separate semantic token stage. Moreover, initializing the super-resolution model from the trained backbone significantly improves convergence and final quality. Taken together, our results suggest that high-quality music generation can be effectively pursued without separating structure and fidelity into heterogeneous representation spaces. Instead, both can be progressively modeled within a unified acoustic-token hierarchy, pointing toward a simpler and more unified path to high-quality music generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。