发现语音大模型解码器有严重冗余，少用60%层仍能保持高精度。
Measuring the Redundancy of Decoder Layers in SpeechLLMs
- 通过剪枝实验发现解码器层数可大幅压缩。
- 7-8B模型仅需60%层数即可保持良好语音识别性能。
- 冗余结构具通用性，适合构建多任务统一语音模型。
语音大语言模型将语音编码器表征输入到通常占总参数90%以上的LLM解码器中。我们研究了语音任务实际需要多少解码器容量。在两个LLM家族和三种规模（1-8B）下，发现解码器冗余主要继承自预训练的LLM：文本与语音输入产生相似的冗余层。通过剪枝解码器层并分析剪枝后恢复能力以提升鲁棒性，结果表明7-8B模型仅需60%解码器层仍能保持良好自动语音识别（ASR）性能，小规模模型则缩减容忍度更低。进一步推广至语音翻译任务，发现相同层在不同语音编码器、任务和语言间均存在冗余，表明存在更全局的冗余结构，支持部署单一剪枝后的多任务语音大模型主干。
原文摘要 · Abstract (English)
Speech Large Language Models route speech encoder representations into an LLM decoder that typically accounts for over 90% of total parameters. We study how much of this decoder capacity is actually needed for speech tasks. Across two LLM families and three scales (1-8B), we show that decoder redundancy is largely inherited from the pretrained LLM: text and speech inputs yield similar redundant blocks. We then measure excess capacity by pruning decoder layers and analysing post-pruning healing to increase robustness. Our findings show that 7-8B models retain good ASR performance with only 60% of decoder layers, and the same trend extends to smaller scales with reduced pruning tolerance. We then generalise to speech translation, and show that the same blocks of layers are redundant across speech encoders, tasks and languages, indicating that a more global redundancy structure exists, enabling a single pruned and multi-tasks SpeechLLM backbone to be deployed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。