用连续音频建模提升生成质量，降低计算成本。
Continuous Audio Language Models
- 用连续帧替代离散令牌，避免有损压缩带来的质量损失。
- 在语音和音乐生成上，质量更高且计算更少，效率优于现有模型。
- 适合需要轻量化高保真音频生成的场景，如端侧部署。
音频语言模型（ALM）通过将音频表示为离散令牌序列，已成为语音与音乐生成的主流范式。然而，与可逆的文本令牌不同，音频令牌来自有损编码器，受限于有限码率。因此，提升音质需生成更多令牌，造成保真度与计算成本间的权衡。本文提出连续音频语言模型（CALM），采用大型Transformer骨干网络在每时刻生成上下文嵌入，并由MLP基于一致性建模生成下一个连续音频帧（来自音频变分自编码器）。该方法规避了有损压缩，实现了更低计算成本下的更高音质。在语音与音乐生成任务上的实验表明，相比最先进离散模型，CALM在效率和保真度上均有提升，支持轻量级、高质量音频生成。样本见hf.co/spaces/kyutai/calm-samples。此外，我们开源了100M参数的Pocket TTS，可在笔记本电脑CPU上超实时运行：github.com/kyutai-labs/pocket-tts。
原文摘要 · Abstract (English)
Audio Language Models (ALM) have emerged as the dominant paradigm for speech and music generation by representing audio as sequences of discrete tokens. Yet, unlike text tokens, which are invertible, audio tokens are extracted from lossy codecs with a limited bitrate. As a consequence, increasing audio quality requires generating more tokens, which imposes a trade-off between fidelity and computational cost. We address this issue by studying Continuous Audio Language Models (CALM). These models instantiate a large Transformer backbone that produces a contextual embedding at every timestep. This sequential information then conditions an MLP that generates the next continuous frame of an audio VAE through consistency modeling. By avoiding lossy compression, CALM achieves higher quality at lower computational cost than their discrete counterpart. Experiments on speech and music demonstrate improved efficiency and fidelity over state-of-the-art discrete audio language models, facilitating lightweight, high-quality audio generation. Samples are available at hf.co/spaces/kyutai/calm-samples. Finally, we release Pocket TTS, an open-source 100M-parameter text-to-speech model that can run faster than real time on a laptop CPU: github.com/kyutai-labs/pocket-tts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。