将多尺度自回归引入频谱图生成，实现高质量音频合成。
MARS: Sound Generation via Multi-Channel Autoregression on Spectrograms
- 将频谱图视为多通道图像，用通道复用降低分辨率但不丢失信息。
- 通过共享分词器从粗到细逐步优化频谱图，提升生成质量。
- 在大规模数据集上表现优于或媲美当前最优方法，适合高保真音频生成。
音频生成研究沿波形与频谱图两种方向发展，涌现出多样化的表示与生成策略。同时，图像合成的进步表明，跨尺度而非跨标记的自回归能提升一致性和细节。基于此，我们提出MARS（Multi-channel AutoRegression on Spectrograms），据我们所知，这是首个将下一尺度自回归建模应用于频谱图领域的模型。MARS将频谱图视为多通道图像，采用通道复用（CMX）策略，在不损失信息的前提下降低空间分辨率。共享分词器在不同尺度间提供一致的离散表示，使基于Transformer的自回归器能高效地从粗到细重构频谱图。在大规模数据集上的实验表明，MARS在多个评估指标上表现可媲美或优于现有最先进方法，建立了一种高效且可扩展的高保真音频生成范式。
原文摘要 · Abstract (English)
Research on audio generation has progressively developed along both waveform-based and spectrogram-based directions, giving rise to diverse strategies for representing and generating audio. At the same time, advances in image synthesis have shown that autoregression across scales, rather than tokens, improves coherence and detail. Building on these ideas, we introduce MARS (Multi-channel AutoRegression on Spectrograms), which, to the best of our knowledge, is the first adaptation of next-scale autoregressive modeling to the spectrogram domain. MARS treats spectrograms as multi-channel images and employs channel multiplexing (CMX), a reshaping strategy that reduces spatial resolution without information loss. A shared tokenizer provides consistent discrete representations across scales, enabling a transformer-based autoregressor to refine spectrograms from coarse to fine resolutions efficiently. Experiments on a large-scale dataset demonstrate that MARS performs comparably or better than state-of-the-art baselines across multiple evaluation metrics, establishing an efficient and scalable paradigm for high-fidelity sound generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。