无需反向传播，用前向计算实现低内存语音模型微调
ZO-ASR: Zeroth-Order Fine-Tuning of Speech Foundation Models without Back-Propagation
- 通过前向传递估算梯度，避免反向传播和激活内存占用
- 在监督领域适配中相对零样本基线降低18.9%词错误率
- 适合资源受限或无法获取梯度的语音模型微调场景
为自动语音识别（ASR）微调预训练语音基础模型通常需要大量GPU内存。本文提出ZO-ASR，一种无需反向传播（BP）和激活内存的零阶（ZO）方法，通过前向传递估计梯度。结合SGD优化器时，ZO-ASR-SGD仅需推理内存即可完成微调。评估涵盖监督与无监督任务：在Whisper-Large-V3的监督域适应中，多查询机制提升鲁棒性，相对零样本基线最高降低18.9%词错误率，优于现有零阶方法；在Wav2Vec2-Base的无监督测试时适应中，性能略低于一阶优化器Adam。该无反向传播方法为计算资源受限或梯度不可用场景下的ASR微调提供了可行方案。
原文摘要 · Abstract (English)
Fine-tuning pre-trained speech foundation models for Automatic Speech Recognition (ASR) is prevalent, yet constrained by substantial GPU memory requirements. We introduce ZO-ASR, a memory-efficient Zeroth-Order (ZO) method that avoids Back-Propagation (BP) and activation memory by estimating gradients via forward passes. When combined with SGD optimizer, ZO-ASR-SGD fine-tunes ASR models using only inference memory. Our evaluation spans supervised and unsupervised tasks. For Supervised Domain Adaptation on Whisper-Large-V3, ZO-ASR's multiple query mechanism enhances robustness and achieves up to an 18.9\% relative Word Error Rate reduction over zero-shot baselines, outperforming existing ZO methods. For unsupervised Test-Time Adaptation on Wav2Vec2-Base, ZO-ASR exhibits moderately lower performance compared to first-order optimizer Adam. Our BP-free approach provides a viable solution for fine-tuning ASR models in computationally resource-constrained or gradient-inaccessible scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。