先训练语音编码器再融合大模型，显著降低计算成本。
Efficient Scaling for LLM-based ASR
- 分阶段训练：先预训练语音编码器，再与大模型融合
- 性能提升21.1%相对词错误率，计算量减少49.9%浮点运算
- 提出可指导部署的语音识别计算规模定律，适合资源受限场景
基于大语言模型（LLM）的自动语音识别（ASR）虽表现优异，但计算开销高。本文通过系统实验发现，先对语音编码器进行预训练，再与LLM整合，相比传统联合微调策略，能显著提升扩展效率。基于此，提出新型多阶段训练方法EFIN（Encoder First Integration），在所有评估策略中持续实现更优性能（相对词错误率降低21.1%），同时大幅降低计算预算（仅需49.9%的浮点运算量）。此外，本文推导出描述ASR错误率与计算量关系的缩放定律，为LLM-ASR的高效扩展提供实用指导。
原文摘要 · Abstract (English)
Large language model (LLM)-based automatic speech recognition (ASR) achieves strong performance but often incurs high computational costs. This work investigates how to obtain the best LLM-ASR performance efficiently. Through comprehensive and controlled experiments, we find that pretraining the speech encoder before integrating it with the LLM leads to significantly better scaling efficiency than the standard practice of joint post-training of LLM-ASR. Based on this insight, we propose a new multi-stage LLM-ASR training strategy, EFIN: Encoder First Integration. Among all training strategies evaluated, EFIN consistently delivers better performance (relative to 21.1% CERR) with significantly lower computation budgets (49.9% FLOPs). Furthermore, we derive a scaling law that approximates ASR error rates as a computation function, providing practical guidance for LLM-ASR scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。