用少量公开语音数据让大模型听懂话,性能接近文本模型。
Closing the Gap Between Text and Speech Understanding in LLMs
- 通过主动选样+跨模态蒸馏,减少语音适配时对文本能力的遗忘
- 仅用公开数据的十分之一,就在多领域评测中达到领先性能
- 适合想低成本提升语音理解能力的研究者和开发者
大型语言模型(LLMs)可被调整以处理语音输入,但其在语音理解任务上的表现始终落后于原始文本模型,甚至低于级联流水线。这种现象被称为文本-语音理解差距:当语音适配的LLM处理语音输入时,性能相比原文本模型处理等效文本显著下降。现有缩小差距的方法或依赖大规模文本语音合成(成本高、依赖合成数据),或依赖不可复现的大规模专有语音数据集。为此,本文分析该差距由两因素导致:(i) 适配过程中对文本能力的遗忘;(ii) 语音与文本间的跨模态错位。基于此,提出SALAD——一种通过主动选择与跨模态蒸馏实现高效对齐的方法,结合针对性合成数据,在减轻遗忘的同时增强对齐。在3B和7B LLM上应用,SALAD仅使用公开语料中不到十分之一的语音数据,即在知识、语言理解与推理等多个领域基准测试中达到与强开源模型相当的性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) can be adapted to extend their text capabilities to speech inputs. However, these speech-adapted LLMs consistently underperform their text-based counterparts--and even cascaded pipelines--on language understanding tasks. We term this shortfall the text-speech understanding gap: the performance drop observed when a speech-adapted LLM processes spoken inputs relative to when the original text-based LLM processes the equivalent text. Recent approaches to narrowing this gap either rely on large-scale speech synthesis of text corpora, which is costly and heavily dependent on synthetic data, or on large-scale proprietary speech datasets, which are not reproducible. As a result, there remains a need for more data-efficient alternatives for closing the text-speech understanding gap. In this work, we analyze the gap as driven by two factors: (i) forgetting of text capabilities during adaptation, and (ii) cross-modal misalignment between speech and text. Based on this analysis, we introduce SALAD--Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation--which combines cross-modal distillation with targeted synthetic data to improve alignment while mitigating forgetting. Applied to 3B and 7B LLMs, SALAD achieves competitive performance with a strong open-weight model across broad-domain benchmarks in knowledge, language understanding, and reasoning, while training on over an order of magnitude less speech data from public corpora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。