解决多语言语音识别中模型乱换语言的问题，提升转录准确率。
Are you speaking my languages? On spoken language adherence in multimodal LLMs
- 用软提示暗示可能的语言，不强制输出单一语言。
- 三种方法使语言错误率降低17%-32%，同时保持整体识别性能。
- 适合需要灵活切换语言的语音应用开发者参考。
基于大语言模型的自动语音识别（ASR）虽支持多语言无缝使用，但模型常误判输出语言，影响转录准确性和下游应用质量。为保留灵活性与语码转换能力，本文提出一种软提示方法，在不严格限制输出的前提下提示潜在口语语言。我们正式定义该问题为语言一致性缺失，并引入新指标量化违规情况。评估三种缓解策略：(1) 零样本提示以在不确定性下提供稳健引导，(2) 监督微调（SFT）提升提示遵循度，(3) 思维链（CoT）推理在解码阶段强化语言一致性。在多种语言上进行对比分析，验证各方法在降低语言错误率的同时维持整体ASR性能的有效性。最后讨论不同计算约束下的权衡，指导策略选择。
原文摘要 · Abstract (English)
While Large Language Model (LLM) based Automatic Speech Recognition (ASR) enables seamless multilingual use, models often misidentify the output language, compromising transcription fidelity and downstream application quality. To preserve flexibility and code-switching capabilities, we propose a soft prompting approach that hints at potential spoken languages without strictly constraining the output. We formally define this challenge as a lack of language adherence, introduce a novel metric to quantify violations, and evaluate three mitigation strategies: (1) zero-shot prompting for robust guidance under uncertainty, (2) supervised fine-tuning (SFT) to improve prompt adherence, and (3) Chain-of-Thought (CoT) reasoning to enforce adherence during decoding. We present a comparative analysis of these methods across multiple languages, evaluating effectiveness in reducing the language violation while maintaining overall ASR performance. Finally, we discuss trade-offs to guide strategy selection under various compute constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。