无需人工标注指令,用自生成数据训练语音大模型,通用性更强。
AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning
- 用冻结的文本大模型自动生成语音任务的监督信号,免去人工标注。
- 仅用25,000小时语音+3,000小时标注语音,在多个基准上达顶尖表现。
- 轻量微调仅改2个投影模块,适合资源有限者快速复现和部署。
将大语言模型(LLM)扩展到语音领域近年备受关注。传统方法通过投影模块连接预训练语音编码器与LLM,再在特定任务的指令数据集上进行训练。然而,构建这类数据耗时费力,且模型在未见任务上泛化能力差。本文提出自生成无指令微调(SIFT)范式,利用冻结的LLM以语音的文本表示为输入生成监督信号,实现最优泛化。基于此,我们构建AZeroS(Auden Zero-instruction-tuned Speech-LLM),在公开语料库中使用约25,000小时带ASR转录的语音和3,000小时带副语言标签的语音进行训练。该模型基于Qwen2.5-7B-Instruct,仅微调两个轻量级投影模块(各2380万参数),保持原始LLM与音频编码器冻结。尽管训练成本极低、数据规模适中,AZeroS在语义与副语言任务基准(包括VoiceBench、AIR-Bench Foundation (Speech) 和 AIR-Bench Chat (Speech))上均达到当前最佳性能。
原文摘要 · Abstract (English)
Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a projection module and trains the resulting model on large-scale, task-specific instruction-tuning datasets. However, curating such instruction-tuning data for specific requirements is time-consuming, and models trained in this manner often generalize poorly to unseen tasks. In this work, we first formulate that the strongest generalization of a speech-LLM is achieved when it is trained with Self-Generated Instruction-Free Tuning (SIFT), in which supervision signals are generated by a frozen LLM using textual representations of speech as input. Our proposed SIFT paradigm eliminates the need for collecting task-specific question-answer pairs and yields the theoretically best generalization to unseen tasks. Building upon this paradigm, we introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora, including approximately 25,000 hours of speech with ASR transcripts and 3,000 hours of speech with paralinguistic labels. Built upon Qwen2.5-7B-Instruct, the model updates only two lightweight projection modules (23.8 million parameters each), while keeping both the LLM and audio encoders frozen. Despite the minimal training cost and modest data scale, AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks, including VoiceBench, AIR-Bench Foundation (Speech), and AIR-Bench Chat (Speech).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。