arXiv:2604.09054cs.SDcs.MM2026-04

用分层自回归模型生成与人声匹配的伴奏,兼顾结构与细节。

HAFM: Hierarchical Autoregressive Foundation Model for Music Accompaniment Generation

论文配图:HAFM: Hierarchical Autoregressive Foundation Model for Music Accompaniment Generation
图 1 · 摘自论文原文
  • 分两阶段编码:50Hz语义令牌+75Hz声学令牌,解耦音乐结构与声音细节
  • 三阶段渐进生成:先构架再粗调后精修,逐步提升伴奏质量
  • 实测音质优于基线18.6%,听感偏好超真人伴奏51.5%

音乐伴奏生成旨在自动创建与给定人声在节奏、和声及音色上协调的乐器伴奏,广泛应用于个性化音乐创作、编曲辅助与音乐教育。现有方法多在符号域或依赖单阶段音频生成框架,普遍存在高层次语义建模不足、声学细节还原弱、条件控制能力差等问题。为此,本文提出HAFM——一种面向人声条件的分层自回归基础模型。该模型采用双速率标记化策略:50 Hz HuBERT语义令牌捕捉高层音乐结构,75 Hz EnCodec声学令牌编码精细声学内容,实现语义与声学表征的显式解耦。在此基础上,设计三阶段级联生成框架,依次生成语义令牌、粗粒度声学令牌与细粒度声学令牌,从全局结构到局部细节逐步优化伴奏。在MUSDB18数据集上的客观评估显示,完整三阶段模型达到FAD分数1.71,相比两阶段基线(FAD=2.10)相对提升18.6%。主观听感测试表明,生成伴奏在一对一比较中获得51.5%的偏好率,显著优于随机基线,在节奏对齐、和声兼容性与整体音乐连贯性方面表现优异。代码与演示已开源。

原文摘要 · Abstract (English)

Music accompaniment generation aims to automatically produce instrumental accompaniments that are rhythmically, harmonically, and timbrally coherent with a given vocal input, with broad applications in personalized music creation, arrangement assistance, and music education. Existing approaches, primarily operating in the symbolic domain or relying on single-stage audio generation frameworks, commonly suffer from insufficient high-level semantic structure modeling, limited acoustic detail reconstruction, and weak conditional controllability. To address these limitations, this paper proposes HAFM, a Hierarchical Autoregressive Foundation Model for vocal-conditioned music accompaniment generation. The model employs a dual-rate tokenization strategy in which $50$ Hz HuBERT semantic tokens capture high-level musical structure and $75$ Hz EnCodec acoustic tokens encode fine-grained acoustic content, enabling explicit disentanglement of semantic and acoustic representations. Building on this foundation, a three-stage cascaded generation framework is designed to progressively generate semantic tokens, coarse acoustic tokens, and fine acoustic tokens, refining the accompaniment from global structure to local detail. . Objective evaluation on the MUSDB18 dataset demonstrates that the full three-stage model achieves a Fr{é}chet Audio Distance (FAD) score of 1.71, representing an 18.6% relative improvement over the two-stage baseline (FAD = 2.10). Subjective listening tests show that the generated accompaniments achieve a 51.5% preference rate against ground-truth accompaniments in head-to-head comparisons, and substantially outperform the random baseline in terms of rhythmic alignment, harmonic compatibility, and overall musical coherence. The source code and demo are available at https://github.com/HackerHyper/HAFM.git.

音乐生成自回归伴奏合成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。