用完整上下文生成音乐,突破传统重建思路,效果超越五个商用系统。
Beyond Reconstruction: Full-Context Generative DiT for Music Generation

- 将音乐生成视为从不完美计划中全序列生成,融合多路音色码本与独立编码的歌词。
- 引入误差匹配干扰条件训练,使模型在噪声预测下仍保持高音质,提升ViSQOL 0.77分。
- 适合追求高保真、长序列音乐生成的研究者与开发者,尤其关注文本与音频对齐的场景。
混合音乐生成器结合自回归语言模型的长程规划能力与基于扩散或流模型的声学渲染高保真度。然而,渲染器在干净的目标解码标记上训练,却在部署时使用不完美的语言模型预测,造成解码器接口暴露偏差。本文不将渲染视为简单重建任务,而是将其建模为从不完美离散计划中的全上下文生成。提出FullDiT,一种条件扩散变换器(DiT),融合八路帧对齐的RVQ流,并对歌词和标题独立编码,同时在完整声学潜在序列上使用非因果自注意力。训练阶段采用误差匹配干扰条件(EMDC),将每码本替换率匹配教师强制下的前1位误差率,并从余弦KNN邻域采样近似错误标记而不改变声学目标。推理阶段采用四路无分类器引导(4-CFG),独立缩放解码器、歌词与标题的引导增量。消融实验表明,EMDC在合成扰动下使ViSQOL提升0.77分,且在固定语言模型标记的非绑定比较中表现更优。进一步消融显示,全曲上下文与渲染端文本条件带来增益。完整系统在18项自动指标中胜过5个商用系统中的15项,并在人工分析音乐带人声排行榜中位列前三。演示页见:https://selinacloudl.github.io/fulldit-demo/
原文摘要 · Abstract (English)
Hybrid music generators combine the long-range planning of an autoregressive language model with the fidelity of a diffusion- or flow-based acoustic renderer. Yet renderers are trained with clean, target-derived codec tokens but deployed with imperfect language-model predictions, creating codecinterface exposure bias. Rather than treating rendering as a simple reconstruction task,we formulate it as full-context generation from an imperfect discrete plan. We introduce FullDiT, a conditional DiT that fuses eight frame-aligned RVQ streams with independently encoded captions and lyrics and uses non-causal self-attention over the complete acoustic latent sequence. During training, Error-Matched Distractor Conditioning (EMDC) matches per-codebook replacement rates to teacher-forced top-1 error rates and samples near-miss tokens from cosine-KNN neighborhoods without changing the acoustic target. At inference, four-way classifier-free guidance (4-CFG) independently scales codec, lyric, and caption guidance increments. Matched ablations show that EMDC improves ViSQOL by 0.77 under synthetic corruption and is clearly preferred in non-tied comparisons with fixed languagemodel tokens. Further ablations show gains from full-song context and renderer-side text conditioning. The complete system outperforms five commercial systems on 15 of 18 automatic metrics and ranks among the top three on the Artificial Analysis Music with Vocals Leaderboard. The demo page is available at https://selinacloudl.github.io/fulldit-demo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。