动态解耦参数,让多语言语音识别模型既共享知识又避免干扰。
Zipper-LoRA: Dynamic Parameter Decoupling for Speech-LLM based Multilingual Speech Recognition
- 通过轻量路由动态调节共享与专属子空间的贡献,实现细粒度参数解耦。
- 在12种语言、资源不均的场景下,低资源语言性能显著提升,优于全共享或全独立方案。
- 适用于实际语音识别系统,对分段和非分段编码器均有效,适合大规模部署。
语音大语言模型(Speech-LLMs)通过将语音编码器与大语言模型对齐,在自动语音识别(ASR)中展现出强大能力。然而,在数据分布不平衡的多语言场景下,如何适配仍具挑战:完全共享的参数高效微调(PEFT)会导致资源少的语言出现负面跨语言干扰,而完全语言特异的微调则限制了跨语言知识迁移,不利于低资源任务。为此,我们提出Zipper-LoRA,一种基于秩级别的动态解耦框架,包含三种变体(静态、硬性、软性),可动态融合共享与语言特异子空间的LoRA更新。通过轻量级语言条件路由机制,该方法在LoRA秩级别上控制各子空间贡献,实现兼容时精细共享、冲突时严格解耦。为增强不均衡数据下的优化稳定性,我们引入两阶段训练策略,配合初始温启动(Initial-B warm start),显著加速收敛。在12种语言混合资源设置下的实验表明,Zipper-LoRA始终优于完全共享和独立基线,尤其在极端低资源情况下表现突出。此外,其优势在分段与非分段编码器配置下均保持稳健,验证了该框架在实际大规模多语言ASR中的可靠性。代码与数据将开源于https://github.com/YuCeong-May/Zipper-LoRA。
原文摘要 · Abstract (English)
Speech Large Language Models (Speech-LLMs) have emerged as a powerful approach for automatic speech recognition (ASR) by aligning speech encoders with large language models. However, adapting these systems to multilingual settings with imbalanced data distributions remains challenging. In such scenarios, a stability-plasticity dilemma often arises: fully shared Parameter-Efficient Fine-Tuning (PEFT) can cause negative inter-lingual interference for under-represented languages, while fully language-specific tuning limits the cross-lingual beneficial knowledge transfer needed for low-resource tasks. To address this, we propose Zipper-LoRA, a novel rank-level decoupling framework with three variants (Static, Hard, and Soft) that dynamically synthesizes LoRA updates from shared and language-specific subspaces. By using a lightweight language-conditioned router, Zipper-LoRA dynamically controls the contribution of each subspace at the LoRA rank level, enabling fine-grained sharing where languages are compatible and strict decoupling when conflicts occur. To further stabilize optimization under imbalanced data, we propose a two-stage training strategy with an Initial-B warm start that significantly accelerates convergence. Experiments on a 12-language mixed-resource setting show that Zipper-LoRA consistently outperforms both fully shared and independent baselines, particularly in extremely low-resource scenarios. Moreover, we demonstrate that these gains are robust across both chunked and non-chunked encoder configurations, confirming the framework's reliability for practical, large-scale multilingual ASR. Our code and data will be available at https://github.com/YuCeong-May/Zipper-LoRA for reproducibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。