用二维频谱图构建音乐分词器，提升自回归生成效果
Modeling Music as a Time-Frequency Image: A 2D Tokenizer for Music Generation
- 将音频频谱转为二维令牌网格，单共享码本减少依赖
- 在数据有限时生成质量优于传统分词方法
- 适合音乐生成、扩散模型等需要结构感知的场景
自回归音乐生成高度依赖音频分词器。现有高保真编码器多采用残差多码本量化，虽保真度高，但序列展平后语言建模受残差层级强依赖影响，易累积误差。本文提出BandTok，一种面向生成的二维梅尔频谱分词器，每帧使用单一共享码本的梅尔频带令牌表示。该设计生成物理可解释的时间-频率令牌网格，令牌间独立性更强，更适配自回归建模。BandTok结合多尺度PatchGAN目标与EMA码本更新提升重建质量，并引入2D旋转位置嵌入（2D RoPE）语言模型，以保留生成过程中的时序与频带结构。实验表明，BandTok在数据受限条件下优于残差码本分词器，生成效果显著。代码与生成演示已公开。
原文摘要 · Abstract (English)
Autoregressive music generation depends strongly on the audio tokenizer. Existing high-fidelity codecs often use residual multi-codebook quantization, which preserves reconstruction quality but complicates language modeling after sequence flattening, as the residual hierarchy imposes strong sequential dependencies and can amplify error accumulation. We propose BandTok, a generation-oriented 2D Mel-spectrogram tokenizer that represents each frame with Mel-frequency band tokens from a single shared codebook. This design yields a physically interpretable time-frequency token grid with a more independent token structure, making it better suited for autoregressive modeling. BandTok improves reconstruction with a multi-scale PatchGAN objective and EMA codebook updates. We further introduce an autoregressive language model with 2D Rotary Position Embedding (2D RoPE) to preserve temporal and frequency-band structure during generation. Experiments show that BandTok improves over residual-codebook tokenizers and achieves strong results in a data-limited setting. The source code and generation demos for this work are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。