用分块策略压缩音频令牌，让语音生成更快更省内存
TLDR: Compressing Audio Tokens for Efficient Autoregressive Text-to-Speech
- 将连续音频令牌分块为小片段，改在块级别进行自回归建模
- 使用4个令牌一组的分块，推理速度提升1.8倍，缓存内存减少75%
- 不替换原有模型，适配现有语音生成系统加速
基于编码器-解码器的自回归语音语言模型通过将语音建模为离散音频令牌序列，在文本转语音任务中取得了优异表现。然而，这种令牌级建模存在结构性效率瓶颈：语音令牌序列远长于文本序列，导致自回归主干需在每个令牌位置执行因果计算，并维护随序列长度增长的键值缓存。本文提出TLDR，一种基于分块的自回归框架，将因果建模从令牌级语音序列转移到块级序列。该方法利用轻量压缩器将连续的编码器令牌聚合成紧凑的潜在块，使用冻结的预训练自回归语音主干（通过LoRA微调）对生成的短块序列进行建模，并通过说话人条件提取器在每块内重建精细语音令牌。实验表明，采用4个令牌为一组的分块方式，TLDR相较基线自回归语音模型实现1.8倍的推理加速，全局键值缓存内存减少高达75%。结果表明，块级全局因果建模是降低预训练编码器-解码器型自回归语音系统推理开销的实用方案，无需替换现有模块。
原文摘要 · Abstract (English)
Codec-based autoregressive (AR) speech language models have achieved strong text-to-speech (TTS) quality by modeling speech as sequences of discrete audio tokens with large pretrained backbones. However, this token-level formulation creates a structural efficiency bottleneck: speech-token sequences are much longer than text sequences, requiring the AR backbone to perform causal computation at every token position and maintain a KV cache that grows with the sequence length. We introduce TLDR, a patch-based autoregressive framework that accelerates codec-based AR-TTS by shifting the causal modeling from token-level speech sequences to patch-level sequences. TLDR groups consecutive codec tokens into compact latent patches using a lightweight compressor, models the resulting shorter patch sequence with a frozen pretrained AR-TTS backbone adapted by LoRA, and reconstructs fine-grained speech tokens within each patch using a speaker-conditioned extractor. With a patch size of 4, TLDR achieves a 1.8x inference speedup over the baseline AR-TTS model and reduces global KV-cache memory by up to 75%. Experimental results indicate that patch-level global causal modeling can be a practical way to reduce the inference cost of pretrained codec-based AR-TTS systems without replacing the existing modules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。