用连续令牌建模音频生成，性能超越现有方法。
Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction
- 用扩散模型处理连续音频令牌，替代传统离散编码。
- 在AudioCaps上FAD和KL分别提升40%和20%。
- 参数量更少，且兼容掩码预测，适合高效音频生成。
自回归的Transformer解码器在大规模语言模型中已成为标准范式，在自然语言处理中取得了显著成功。将这一范式扩展到音频领域面临独特挑战，因其本质为连续信号。本文研究无需离散令牌的因果语言模型进行音频生成，利用逐令牌扩散来建模下一个连续值令牌的连续分布。该方法相比先前的离散方案AudioGen，在AudioCaps数据集上分别实现了20%和40%的弗雷切特音频距离（FAD）与相对熵（KL）改善。此外，我们提出一种新颖的掩码下一段预测任务，将其融入因果语言模型框架。在AudioCaps上，该创新使Base（285M）和Large（1B）版本分别实现41%和33%的相对FAD提升，性能媲美当前最先进扩散模型。同时，我们的模型仅需193M（Base）和462M（Large）参数即可达成此效果。
原文摘要 · Abstract (English)
Autoregressive next-token prediction with the Transformer decoder has become a de facto standard in large language models (LLMs), achieving remarkable success in Natural Language Processing (NLP) at scale. Extending this paradigm to audio poses unique challenges due to its inherently continuous nature. We research audio generation with a causal language model (LM) without discrete tokens. We leverage token-wise diffusion to model the continuous distribution of the next continuous-valued token. Our approach delivers significant improvements over previous discrete solution, AudioGen, achieving 20% and 40% relative gains on AudioCaps in Frechet Audio Distance (FAD) and Kullback-Leibler (KL) divergence, respectively. Additionally, we propose a novel masked next-token prediction task that incorporates masked prediction into the causal LM framework. On AudioCaps, the innovation yields 41% and 33% relative FAD improvements over AudioGen Base (285M) and AudioGen Large (1B) models, respectively, and is on par with the state-of-the-art (SOTA) diffusion models. Furthermore, we achieve these results with significantly fewer parameters -- 193M for our Base and 462M for our Large models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。