分析语音模型文本能力下降原因，提出保留关键参数分布的修复方法
Understanding Textual Capability Degradation in Speech LLMs via Parameter Importance Analysis
- 通过参数重要性分析发现语音微调破坏了文本推理所需参数分布
- 使用分层学习率或LoRA可更好保持文本能力，提升语音问答性能
- 为修复策略提供理论依据，适合研究多模态模型鲁棒性的学者
将语音引入大语言模型虽拓展了能力，但常导致其核心文本能力下降，限制了预训练文本知识的利用。本文聚焦广泛使用的编码器-适配器范式，提出基于参数重要性估计的分析框架，揭示语音微调引发文本重要性分布偏移：各层对文本推理至关重要的参数分配被打乱。基于此，我们探究两种缓解策略——分层学习率调度和低秩适配（LoRA），均旨在保持原有参数分布。实验表明，二者在维持文本能力方面优于全量微调，同时提升下游语音问答表现。此外，分析为所提策略的有效性提供了原则性解释，将其优势与大语言模型中文本知识的结构特性关联。
原文摘要 · Abstract (English)
The integration of speech into Large Language Models (LLMs) has substantially expanded their capabilities, but often at the cost of weakening their core textual competence. This degradation limits the ability of speech-enabled LLMs to fully exploit their pre-trained text-based knowledge. In this work, we analyze the underlying mechanisms of this issue through a focused study of the widely used encoder-adaptor paradigm. We propose an analytical framework based on parameter importance estimation, which reveals that fine-tuning for speech introduces a textual importance distribution shift: the layer-wise allocation of parameters critical to textual reasoning is disrupted. Building on this insight, we investigate two mitigation strategies: layer-wise learning rate scheduling and Low-Rank Adaptation (LoRA), both aim to preserve the original parameter distribution. Experimental results show that both approaches better maintain textual competence than full fine-tuning, while also improving downstream spoken question answering performance. Furthermore, our analysis offers a principled explanation for the effectiveness of the proposed mitigation strategies, linking their benefits to the structural properties of textual knowledge in LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。