通过分阶段提前退出机制,加速语音语言模型推理且保持音质。
SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Models
- 根据语音模态特点设计分层提前退出策略,固定中间层退出多数语音位置。
- 在多个数据集上减少平均解码深度11%(Step-Audio-2-mini)至5%(GLM-4-Voice),准确率下降≤0.82%。
- 针对语音特征定制方案,优于文本大模型通用退出方法,适合语音生成任务优化。
交错式语音语言模型(SLMs)交替生成文本与语音标记,但每步都进行完整Transformer解码代价高昂,尤其在长语音序列下。本文提出SPAR-K,一种模态感知的提前退出框架,旨在加速交错式SLM推理同时保持听觉质量。SPAR-K引入语音交替深度调度:多数语音位置在固定中间层提前退出,周期性地进行全深度“刷新”以缓解因提前退出导致的分布偏移。我们在Step-Audio-2-mini和GLM-4-Voice上评估该框架,在涵盖推理、事实问答和对话的四个数据集上,以语音识别转录准确率(WER)和感知质量(MOS)为指标。实验表明,SPAR-K在最多0.82%准确率损失下,将Step-Audio-2-mini的平均语音解码深度降低11%,将GLM-4-Voice降低5%,且对MOS和WER影响可忽略,无额外计算开销。进一步证明,广泛应用于文本大模型的置信度提前退出策略对语音模型不适用,凸显语音标记独特的统计特性需专用提前退出设计。
原文摘要 · Abstract (English)
Interleaved spoken language models (SLMs) alternately generate text and speech tokens, but decoding at full transformer depth for every step becomes costly, especially due to long speech sequences. We propose SPAR-K, a modality-aware early exit framework designed to accelerate interleaved SLM inference while preserving perceptual quality. SPAR-K introduces a speech alternating-depth schedule: most speech positions exit at a fixed intermediate layer, while periodic full-depth "refresh" steps mitigate distribution shift due to early exit. We evaluate our framework using Step-Audio-2-mini and GLM-4-Voice across four datasets spanning reasoning, factual QA, and dialogue tasks, measuring performance in terms of ASR transcription accuracy and perceptual quality. Experimental results demonstrate that SPAR-K largely preserves question-answering accuracy with a maximum accuracy drop of 0.82\% while reducing average speech decoding depth by up to 11\% on Step-Audio-2-mini and 5\% on GLM-4-Voice, both with negligible changes in MOS and WER and no auxiliary computation overhead. We further demonstrate that confidence-based early exit strategies, widely used in text LLMs, are suboptimal for SLMs, highlighting that the unique statistical nature of speech tokens necessitates a specialized early exit design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。