用语音合成生成卢森堡语问答数据,让低资源语言也能用语音大模型。
LuxSQA: Ask Me in Luxembourgish with TTS-Augmented Spoken Question Answering

- 用TTS合成卢森堡语语音问题,搭配已有文本答案构建训练数据。
- 23万条多语音合成数据比4.8万条单源数据提升明显,最佳模型准确率超基准3.7%。
- 合成语音质量不能只看听感,必须评估其对问答任务的实际帮助。
语音问答(SQA)主要集中在高资源语言和精心录制的语音上,限制了语音大模型在低资源场景的应用。本文探讨文本到语音(TTS)能否在无需大规模人工录音问答语料的情况下,为卢森堡语SQA提供任务特定的训练数据。基于现有文本问答资源,我们将问题翻译为卢森堡语,使用多种TTS系统合成语音问题,并与文本答案配对。我们训练了一种参数高效、类似SLAM的架构,将冻结的Whisper编码器与冻结的多语言LLM后端通过可学习投影层和LoRA适配器连接。对比MMS-TTS、Qwen3-TTS和OmniVoice等变体,包括约4.8万条单源语料和约23万条多源混合语料。在LLAMA-LB-Test上,针对两位真实卢森堡语使用者的测试条件进行评估,结果显示多源及语音风格设计的合成训练配置表现最优。结果还表明,无参考的TTS质量评分无法单调预测下游问答性能,说明合成语音必须作为任务相关的训练数据来评估,而不仅是自然发音的音频。
原文摘要 · Abstract (English)
Spoken Question Answering (SQA) remains largely focused on high-resource languages and carefully recorded speech, limiting the reach of speech-LLM methods in low-resource settings. This paper investigates whether text-to-speech (TTS) can provide task-specific training data for Luxembourgish SQA without requiring a large human-recorded QA corpus. Starting from existing text-based QA resources, we translate questions into Luxembourgish, synthesize spoken questions with multiple TTS systems, and pair them with textual answers. We train a parameter-efficient SLAM-style architecture that connects a frozen Whisper encoder to frozen multilingual LLM backends through a learned projector and LoRA adapters. We compare MMS-TTS, Qwen3-TTS, and OmniVoice variants, including single-source corpora of about 48k questions and a 4TTS multi-source mix of approximately 230k questions. Evaluation on LLAMA-LB-Test with two real Luxembourgish speaker conditions shows that multi-source and voice-design-based synthetic training configurations yield the strongest SQA performance. The results also show that no-reference TTS quality scores do not monotonically predict downstream QA performance, indicating that synthetic speech must be evaluated as task-specific training data rather than only as natural-sounding audio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。