通过压缩语音的键值缓存，实现高效语音大模型推理。
Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs
- 在模型内部对语音键值缓存进行可学习池化，保留语义信息
- 压缩后语音序列接近文本粒度，性能持平或略优
- 适用于长语音场景，推理速度提升1.49倍以上
语音大语言模型通常将语音编码为远超文本长度的序列，导致自回归解码效率低下。常见方法是在适配器层提前下采样以去除时间冗余，但可能丢失无法恢复的细粒度信息。本文提出SpeechKV，对语音令牌的键值缓存进行可学习池化。该设计使模型在内部融合语音与文本的同时直接加速解码。在71,000小时语音数据上训练的SpeechKV，将语音压缩至接近文本级别粒度，性能与未压缩基线相当甚至略有提升，在跨域实体识别上相对提升6.6%，在OpenASR上提升2.3%，同时解码速度至少提升1.49倍，且随音频长度增长而进一步提升。
原文摘要 · Abstract (English)
Speech large language models (Speech LLMs) typically encode speech into sequences far longer than text, creating a major efficiency bottleneck during autoregressive decoding. A common remedy is to compress the speech sequence at the adapter level to remove temporal redundancy before it enters the LLM; however, such early downsampling risks discarding fine-grained information that cannot be recovered. We propose SpeechKV, which applies a learned pooling to the KV cache of speech tokens inside the LLM. This design allows the LLM to fuse speech and text internally while directly accelerating decoding. Trained on 71K hours of speech data, SpeechKV compresses the speech to approximately text-level granularity yet maintains performance on par with or even slightly better than the uncompressed baseline, with relative gains of 6.6% on out-of-domain entity recognition and 2.3% on OpenASR, while delivering at least 1.49 times decoding speedup that scales with audio length.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。