SpeechMapper让语音转文本嵌入更高效，无需大量训练就能通用。
SpeechMapper: Speech-to-text Embedding Projector for LLMs
- 先低成本预训练语音嵌入，再快速微调接入大模型
- 任务无关微调下性能媲美顶尖语音大模型
- 适合资源有限但需通用语音理解的场景
当前语音大模型通过投影层将语音基础模型与大语言模型连接，并在语音指令数据上联合训练，但计算成本高且易过拟合。本文提出SpeechMapper，一种低成本的语音到大模型嵌入训练方法，可缓解过拟合，提升模型鲁棒性与泛化能力。该模型先在廉价硬件上无大模型参与地预训练，再通过仅1000步的指令微调阶段高效接入目标大模型。在语音翻译和语音问答任务中，我们验证了其预训练模块的多功能性，展示了任务无关微调、基于ASR的适配策略及任务特定微调三种方式。在任务无关设置下，SpeechMapper在IWSLT25上的表现媲美最佳指令跟随语音大模型，尽管从未在这些任务上训练；在任务特定设置下，其在多个数据集上超越该模型，且所需数据与计算更少。整体而言，SpeechMapper提供了一种高效、可扩展的语音-大模型融合方案，无需大规模指令微调。
原文摘要 · Abstract (English)
Current speech LLMs bridge speech foundation models to LLMs using projection layers, training all of these components on speech instruction data. This strategy is computationally intensive and susceptible to task and prompt overfitting. We present SpeechMapper, a cost-efficient speech-to-LLM-embedding training approach that mitigates overfitting, enabling more robust and generalizable models. Our model is first pretrained without the LLM on inexpensive hardware, and then efficiently attached to the target LLM via a brief 1K-step instruction tuning (IT) stage. Through experiments on speech translation and spoken question answering, we demonstrate the versatility of SpeechMapper's pretrained block, presenting results for both task-agnostic IT, an ASR-based adaptation strategy that does not train in the target task, and task-specific IT. In task-agnostic settings, Speechmapper rivals the best instruction-following speech LLM from IWSLT25, despite never being trained on these tasks, while in task-specific settings, it outperforms this model across many datasets, despite requiring less data and compute. Overall, SpeechMapper offers a practical and scalable approach for efficient, generalizable speech-LLM integration without large-scale IT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。