用语言感知机制提升多语言语音大模型的指令遵循能力。
Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision
- 引入查询库和门控网络，实现多语言语境下的智能注意力选择。
- 在多语言指令遵循任务上比基线提升14%，在语音问答上领先32%。
- 构建Audio-MLQA基准，适合评估跨语言语音理解模型性能。
能够理解并响应多种语言指令的语音大模型对真实交互极具价值，但其训练依赖大量特定任务的语音数据，难以通过监督微调实现。尽管现有基于蒸馏的方法仅使用标注的语音识别数据即可训练出高性能的英文语音大模型，但由于共享投影器中的语言干扰，模型在扩展至多语言场景时表现下降。本文提出语言感知蒸馏方法，通过查询库与门控网络结合Q-Former投影器，动态选择或混合查询令牌以缓解语言混淆。该方法在多语言指令遵循任务上相较基线提升14%。此外，我们构建了Audio-MLQA——一个基于MLQA的高质量多语言语音问答基准，采用合成语音提问。最优模型在Audio-MLQA上相比现有语音大模型基线提升32%。
原文摘要 · Abstract (English)
Speech Large Language Models (LLMs) that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。