用近似错误样本增强模型，提升多语言混杂语音识别准确率。
Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition
- 基于关键位置检测生成逼真的近似错误语音样本
- 在中英、越英混杂数据集上错误率降低超2%
- 适合需要高鲁棒性多语言语音识别的场景
语音识别中的语言混杂（代码切换）问题依然严峻。本文提出一种关注关键位置（POI）的对比训练框架，聚焦于代码切换的关键区域。首先通过文献中的关键位置检测方法识别出语言切换段落，然后在ASR的候选输出中扰动这些关键位置，并利用大语言模型扩展候选集，生成声学上合理且接近错误的近似样本。通过声学、音素和文本三重约束筛选保留困难但合理的负样本。最后，使用LoRA微调Whisper-small模型，结合加权交叉熵锚点目标与多负样本对比排序损失。在CS-FLEURS（中文-英文）和ViMedCSS（越南语-英文）数据集上的实验表明，相比标准的LoRA微调，整体错误率和针对代码切换的错误率均显著降低超过2%。
原文摘要 · Abstract (English)
Code-switching (CS), the alternation between multiple languages within a single utterance, remains challenging for Automatic Speech Recognition (ASR). To address this issue, we propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions. We first identify CS spans by adopting POI detection method from literature, then construct acoustically plausible near-miss hypotheses by perturbing POIs in ASR N-best outputs and expanding candidates with a large language model. Hard but plausible negatives are retained through filtering with acoustic, phonemic, and textual constraints. Finally, we fine-tune Whisper-small with LoRA using a POI-weighted cross-entropy anchor objective together with a multi-negative contrastive ranking loss. Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates compared to standard LoRA fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。