通过结构化SVD实现语音识别的高效微调，平衡不同模块参数分配。
SSVD-O: Parameter-Efficient Fine-Tuning with Structured SVD for Speech Recognition
- 基于输入与输出特征空间的内外变换组合，动态分配参数预算。
- 在0.1B至2B模型上，性能接近全量微调且遗忘更少。
- 适合资源受限下跨语种、儿童语音等场景的语音模型适配。
参数高效微调（PEFT）是将大型语音基础模型适应新领域的可扩展方法。尽管LoRA及其先进变体降低了适应成本，但通常在模型子空间中均匀分配参数，限制了其在语音任务中的效率与可扩展性。本文基于前期工作，提出结构化SVD引导的外层微调方法（SSVD-O），结合输入声学特征空间相关的内变换与输出语义特征空间相关的外变换，实现可扩展且均衡的适配。我们首次系统分析了自动语音识别（ASR）中PEFT的参数预算分配策略，并研究了资源受限下的学习与遗忘权衡。在ESPnet框架下，针对儿童语音和区域口音等域偏移任务，对0.1B至2B规模模型进行了实验，对比了LoRA、DoRA、PiSSA和SSVD。结果表明，SSVD-O始终缩小与全量微调的性能差距，同时提升泛化能力并缓解灾难性遗忘。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning (PEFT) is a scalable approach for adapting large speech foundation models to new domains. While methods such as LoRA and its state-of-the-art variants reduce adaptation costs, they typically allocate parameters uniformly across model subspaces, which limits their efficiency and scalability in speech applications. Building on our prior work, this paper introduces SSVD-Outer (SSVD-O), an extension of the structured SVD-guided (SSVD) fine-tuning method. SSVD-O combines input acoustic feature space-associated inner transformations with output semantic feature space-associated outer transformations to enable scalable and balanced adaptation. We conduct the first systematic analysis of parameter budget allocation across model subspaces in PEFT for automatic speech recognition (ASR), and investigate the trade-off between learning and forgetting under constrained resources. SSVD-O is benchmarked against LoRA, DoRA, PiSSA, and SSVD on domain-shifted ASR tasks, including child speech and regional accents, across model scales from 0.1B to 2B within the ESPnet framework. Experimental results show that SSVD-O consistently narrows the performance gap to full fine-tuning while improving generalization and mitigating catastrophic forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。