arXiv:2608.04378cs.SDcs.LG2026-08

用自监督学习让音乐生成模型理解乐句与和声结构,支持人机协作创作。

Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation

论文配图:Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation
图 1 · 摘自论文原文
  • 通过分层自监督训练,从无标签钢琴卷数据中自动学习音乐结构
  • 在无和弦标注下实现关键识别准确率从16%提升至70%,和弦恢复达54%
  • 可快速生成建议(2.8秒),支持图形化掩码补全,适合交互式音乐创作

协作音乐代理需要具备既能理解又能生成的丰富内部表示,同时保持人类主导权。本文提出一种用于符号化音乐的分层自监督“世界模型”:一个255万参数的Swin V2编码器,基于钢琴卷图像以JEPA式目标(音高与时间平移等变性、掩码嵌入预测、分布正则化)训练,无需标签或音乐理论词汇。探测冻结嵌入显示,音乐属性的可解码层级与其音乐时间尺度一致:乐句边界在粗粒度层级读取,音符密度与和声细节在细粒度层级。时间与乐句结构仅由自监督目标自发涌现,而和声内容需显式引入;添加小规模和弦监督头后,联合和弦恢复准确率从0.18提升至0.54,从未受监督的关键检测准确率从0.16提升至0.70。遵循表示自编码器范式,采用条件流匹配模型替代训练解码器,在主成分分析降维后的条件空间中进行像素级生成:对目标窗口的像素F1达0.996;同级别的条件丢弃策略控制变化范围,也实现了无需专用采样器的图形化掩码补全。该流程可在CPU上2.8秒内生成建议,或在Apple MPS上0.6秒完成,已在实时交互演示中验证。结合基于大语言模型的决策模块,这些能力构成了服务于而非取代人类创造力的协作音乐生成核心。

原文摘要 · Abstract (English)

Collaborative music agents need internal representations rich enough to support both understanding and generation, yet flexible enough for a workflow where the human retains agency. We present a hierarchical self-supervised ``world model'' for symbolic music: a 2.55M-parameter Swin V2 encoder trained on MIDI piano-roll images with JEPA-style objectives (pitch- and time-shift equivariance, masked embedding prediction, and a distributional regularizer), using no labels and no music-theory vocabulary. Probing the frozen embeddings shows that the level at which a musical property becomes decodable tracks its musical time scale: phrase boundaries are read off the coarsest levels, note density and harmonic detail off the finest. Temporal and phrase structure emerge from the self-supervised objectives alone, while harmonic content must be asked for; a small chord-supervision head raises joint chord recovery from .18 to .54, and key detection, which is never supervised, from .16 to .70. Following the Representation AutoEncoder paradigm, a conditional flow-matching model stands in for a trained decoder, flowing in pixel space from PCA-reduced conditioning: it reproduces a target window at pixel F1 $0.996$, and the same per-level conditioning dropout that controls how far variations stray also enables graphical prompting for masked inpainting with no inpainting-specific sampler. The pipeline runs on CPU producing a suggestion in $2.8$ s, or $0.6$ s on Apple MPS, which we demonstrate in a live interactive demo. In concert with an LLM-based brain, these capabilities supply the core of a collaborative music creation agent in service of, rather than in place of, human agency.

音乐生成自监督学习人机协作世界模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。