融合连续谱图与离散音符，提升语音音乐生成的预测精度
Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music
- 用连续谱图和离散音符联合建模，兼顾信息完整与生成效率
- 在语音音乐任务上，困惑度和负对数似然均优于纯离散模型
- 适合需要长上下文生成的语音音乐应用开发者
我们提出 WHISPER-GPT：一种面向语音和音乐的生成式大语言模型，能够同时处理连续音频表示（如频谱图）和离散音频标记。近年来，基于神经压缩算法（如 ENCODEC）生成的离散音频标记在语音与音乐生成中广泛应用。然而，该方法在高保真生成架构中面临上下文长度爆炸的问题，需对多频段音频内容进行全量建模。通过结合连续音频表示与离散声学标记，我们实现了两者的最优平衡：单个标记即可包含特定时间点所需的全部音频信息，同时保持语言模型对后续标记的预测能力，从而支持采样等优势。实验表明，相较于纯离散标记的大型语言模型，本架构在语音与音乐任务上的困惑度与负对数似然得分均有显著提升。
原文摘要 · Abstract (English)
We propose WHISPER-GPT: A generative large language model (LLM) for speech and music that allows us to work with continuous audio representations and discrete tokens simultaneously as part of a single architecture. There has been a huge surge in generative audio, speech, and music models that utilize discrete audio tokens derived from neural compression algorithms, e.g. ENCODEC. However, one of the major drawbacks of this approach is handling the context length. It blows up for high-fidelity generative architecture if one has to account for all the audio contents at various frequencies for the next token prediction. By combining continuous audio representation like the spectrogram and discrete acoustic tokens, we retain the best of both worlds: Have all the information needed from the audio at a specific time instance in a single token, yet allow LLM to predict the future token to allow for sampling and other benefits discrete space provides. We show how our architecture improves the perplexity and negative log-likelihood scores for the next token prediction compared to a token-based LLM for speech and music.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。