发现语音模型深层冗余，用相似性合并令牌提升效率
Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models
- 通过层间干预发现深层语音表示高度冗余
- 压缩后推理耗能降27.48%，首词生成速度提10%以上
- 无需训练的合并方法，适合长语音部署场景
大型语音语言模型（LSLM）通常以高令牌率（tokens/s）运行以保证声学保真度，但导致序列长度远超语义内容，带来巨大的推理开销。本文通过逐层虚拟干预实证重审了细粒度令牌处理的必要性。结果揭示出结构化冗余层级：浅层保留关键声学信息，深层则存在极端冗余，可大幅压缩。受此启发，我们提出无需训练的相似性聚合方法——Affinity Pooling。在输入层与深层有策略地应用该方法，可在不损失语义的前提下压缩语音表示。三类任务的广泛评估表明，该方法将预填充阶段浮点运算量减少27.48%，同时保持优异精度。实际部署验证显著提升效率：长语音场景下内存节省约1.7倍，首词生成时间加快约1.1倍。研究挑战了为每个语音令牌保持独立表征的必要性，为LSLM效率优化提供新视角。
原文摘要 · Abstract (English)
Large Speech Language Models (LSLMs) typically operate at high token rates (tokens/s) to ensure acoustic fidelity, yet this results in sequence lengths that far exceed the underlying semantic content, incurring prohibitive inference costs. In this paper, we empirically revisit the necessity of such granular token-level processing. Through layer-wise oracle interventions, we unveil a structured redundancy hierarchy: while shallow layers encode essential acoustic details, deep layers exhibit extreme redundancy, allowing for aggressive compression. Motivated by these findings, we introduce Affinity Pooling, a training-free, similarity-based token merging mechanism. By strategically applying this method at both input and deep layers, we effectively compress speech representations without compromising semantic information. Extensive evaluations across three tasks demonstrate that our approach reduces prefilling FLOPs by 27.48\% while maintaining competitive accuracy. Practical deployment further confirms significant efficiency gains, yielding up to $\sim$1.7$\times$ memory savings and $\sim$1.1$\times$ faster time-to-first-token on long utterances. Our results challenge the necessity of fully distinct token representations, providing new perspectives on LSLM efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。