用强化学习让语音模型更准识别混语，少数据也能达到顶尖效果。
Reinforcement Learning for Data-Efficient Code-Switched ASR
- 通过奖励机制优化混语语音识别的解码策略
- 仅用10%数据就达到全量数据微调效果，错误率显著降低
- 适合资源有限但需高精度混语识别的研究与应用
音频-语言模型可提示处理混语语音，但其解码未针对混语优化，常在语言边界失败。我们提出一种可验证奖励的强化学习方法（RLVR），结合误差率奖励与书写系统保真度奖励，采用两阶段草稿-精炼流程，实现数据高效适配。以Qwen2-Audio为基准，在10个语种对上仅使用语音合成的混语数据训练，仅10%数据量即达到全量数据微调的LoRA性能，尤其在语言类型差异大的组合中提升最明显。误差率奖励消除翻译错误，保真度奖励减少书写系统混淆且不损害性能。该方法零样本迁移至人工录制的混语数据集仍有效。
原文摘要 · Abstract (English)
Audio-language models can be prompted for code-switched speech, but their decoding is not optimized for code-switching and often fails at language boundaries. We propose a practical reinforcement learning with verifiable rewards recipe for data-efficient adaptation of audio-language models to code-switched ASR using group relative policy optimization, combining an error rate reward with a script fidelity reward that penalizes wrong writing systems and a two-pass draft-and-refinement procedure. Using Qwen2-Audio as a reproducible testbed across 10 language pairs, training on only TTS code-switched speech, we show that RLVR with 10% of the data matches LoRA supervised fine-tuning trained on the full dataset, with the largest gains on typologically distant pairs. The error rate reward eliminates translation errors while the script fidelity reward separately reduces script contamination without degradation. These gains transfer zero-shot to a human-recorded code-switching corpus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。