用轻量适配器让大模型理解语音，低资源下多任务表现大幅提升
SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings
- 用参数高效适配器将语音特征转为文本令牌，实现端到端多任务理解
- 在低资源下，语音识别错误率降26%，实体识别和情感分析准确率分别提升6.3%和32%
- 适合语音多任务研究、资源受限场景的开发者使用
将语音编码器与大语言模型结合需大量数据与算力，而实际应用常受限于标注资源不足。为此，我们提出一种参数高效的适配器，将语音嵌入转换为大语言模型兼容的标记，聚焦端到端自动语音识别（ASR）、命名实体识别（NER）和情感分析（SA）。为降低标注成本，采用基于大语言模型的合成数据标注技术。所提适配器仅使用7倍更少的可训练参数，在LibriSpeech ASR任务上实现26%相对词错误率降低，在NER任务上相对F1提升6.3%，在SA任务上相对F1提升32%。进一步结合分类器正则化与低秩适配（LoRA）优化大模型，使口语理解评估（SLUE）得分分别提升6.6%和9.5%。
原文摘要 · Abstract (English)
While integrating speech encoder with LLM requires substantial data and resources, use cases face limitations due to insufficient availability. To address this, we propose a solution with a parameter-efficient adapter that converts speech embeddings into LLM-compatible tokens, focusing on end-to-end automatic speech recognition (ASR), named entity recognition (NER), and sentiment analysis (SA). To reduce labeling costs, we employ an LLM-based synthetic dataset annotation technique. The proposed adapter, using 7x fewer trainable parameters, achieves significant performance gains: a 26% relative Word Error Rates (WER) improvement on the LibriSpeech ASR task, a 6.3% relative F1 score increase on the NER task, and a 32% relative F1 score boost on the SA task. Moreover, using advanced techniques such as adding a classifier regularizer and optimizing the LLM with Low-Rank Adaptation (LoRA) yields notable performance gains, with Spoken Language Understanding Evaluation (SLUE) score improvement of 6.6% and 9.5%
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。