让语音大模型一次生成多个词，推理速度提升3倍以上
Faster Speech-LLaMA Inference with Multi-token Prediction
- 一次解码预测多个词，打破逐词生成的限制
- 在多个评测集上将解码调用次数减少约3.2倍，准确率不变或更高
- 提出前缀式束搜索，支持高效最小词错误率训练
大型语言模型（LLMs）已能处理多种任务，包括多模态输入。将语音编码器与LLaMA等模型结合，并在配对数据上训练，可使仅解码器模型具备语音识别（ASR）能力，称为Speech-LLaMA。然而，由于自回归解码的串行特性及较大的解码器，Speech-LLaMA推理耗时较长。本文提出通过单次解码步骤预测多个词来加速推理，探索了多种支持该机制的模型架构，并采用基于阈值和验证的推理策略。此外，提出一种前缀式束搜索解码方法，实现此类模型的高效最小词错误率（MWER）训练。在多个公开基准上评估显示，该方法使解码调用次数减少约3.2倍，同时保持或提升了词错误率（WER）性能。
原文摘要 · Abstract (English)
Large language models (LLMs) have become proficient at solving a wide variety of tasks, including those involving multi-modal inputs. In particular, instantiating an LLM (such as LLaMA) with a speech encoder and training it on paired data imparts speech recognition (ASR) abilities to the decoder-only model, hence called Speech-LLaMA. Nevertheless, due to the sequential nature of auto-regressive inference and the relatively large decoder, Speech-LLaMA models require relatively high inference time. In this work, we propose to speed up Speech-LLaMA inference by predicting multiple tokens in the same decoding step. We explore several model architectures that enable this, and investigate their performance using threshold-based and verification-based inference strategies. We also propose a prefix-based beam search decoding method that allows efficient minimum word error rate (MWER) training for such models. We evaluate our models on a variety of public benchmarks, where they reduce the number of decoder calls by ~3.2x while maintaining or improving WER performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。