给文本大模型加新层，用语音数据微调，兼顾文本与语音能力。
Adapting Text LLMs to Speech via Multimodal Depth Up-Scaling
- 在冻结的文本模型中插入新层，仅训练新增部分处理语音。
- 语音识别效果接近全量微调，文本能力退化减少75%以上。
- 结合专用语音架构，参数少60%仍保持顶尖语音性能。
通过在预训练文本大模型上持续预训练语音数据来适配为语音语言模型具有前景，但常导致原有文本能力下降。本文提出多模态深度扩展（Multimodal Depth Upscaling），在冻结的文本大模型中插入新Transformer层，仅训练新增层处理语音数据。在48,000小时英语自动语音识别（ASR）数据上对SmolLM2-360M和SmolLM2-1.7B进行实验，结果表明该方法实现的语音识别性能接近全量微调，而文本退化程度远低于全量微调和低秩适配（LoRA）。进一步将专为语音识别设计的E-Branchformer作为插入层，使大模型的语音识别表现达到或超过全量微调水平，同时文本退化降低超过75%，可训练参数减少60%。
原文摘要 · Abstract (English)
Adapting pre-trained text Large Language Models (LLMs) into Speech Language Models (Speech LMs) via continual pretraining on speech data is promising, but often degrades the original text capabilities. We propose Multimodal Depth Upscaling, an extension of an emerging strategy in continual LLM pre-training, where new transformer layers are inserted into a frozen text LLM and only the added layers are trained on speech data. Experiments with SmolLM2-360M and SmolLM2-1.7B on 48k hours of English Automatic Speech Recognition (ASR) data show that depth up-scaling achieves ASR comparable to full fine-tuning while causing far less text degradation than both full fine-tuning and Low-Rank Adaptation (LoRA). We further show that incorporating E-Branchformer, an architecture designed for speech recognition, as the inserted layers achieves ASR that matches or surpasses full fine-tuning on the larger model while reducing text degradation by over 75% with 60% fewer trainable parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。