用直接偏好优化让语音大模型正确识别中英混杂语音
Direct Preference Optimization for English-Mandarin Code-Switching Speech Recognition in Audio LLMs
- 用偏好对训练模型，让其学会保留中英混合语序而非翻译
- 在10万组数据上训练后，识别错误率降低最高达89.6%
- 适合需要准确处理中英混杂语音的语音识别场景
语音大语言模型（Audio LLMs）虽具备强大多语言能力，但在处理中英混杂语音时仍存在系统性失败。针对英语-中文混用场景，我们识别出三种失败模式：语言遗漏、翻译替代转录、以及幻觉。本文采用直接偏好优化（DPO）进行模型对齐，构建偏好对：优选响应保留混合语言内容，次选响应模拟上述失败模式。在10万对样本（570小时）上训练三个Audio LLM，观察到行为显著转变——模型在转录提示下更倾向于保留语言组成而非翻译。该对齐方法使内分布误差率（MER）降低最高达89.6%，外分布降低20.0%。结果表明，DPO可有效引导多语言Audio LLM实现正确的代码混杂语音转录行为。
原文摘要 · Abstract (English)
Audio large language models (Audio LLMs) exhibit systematic failures in transcribing code-switching speech despite strong multilingual capabilities. Focusing on English-Mandarin, we identify three failure modes: language omission, translation-instead-of-transcription, and hallucination. We apply Direct Preference Optimization (DPO) to align models, constructing preference pairs in which chosen responses preserve mixed-language content while rejected responses mimic failure patterns. Training three Audio LLMs on 100K pairs (570 hours), we observe consistent behavioral shifts: models learn to preserve language composition rather than translating when prompted for transcription. This alignment yields MER reductions up to 89.6% (in-distribution) and 20.0% (out-of-distribution). Our findings suggest DPO can effectively elicit correct code-switching transcription behavior from multilingual Audio LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。