提出新型软最大化方法，提升声调与非声调语言的多语言语音识别准确率。
Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages
- 将带声调元音设为子类、基础元音为大类，实现跨语言共享
- 在中文和英文数据上分别降低8.4%、17.5%的音素错误率
- 适合需要高精度多语言语音识别的研究者与开发者
基于音素的多语言自动语音识别（ASR）相比语言特定子词建模能更直接地共享声学信息。然而，在声调与非声调语言联合训练时，标注粒度不匹配：声调语言标注带声调的元音，而非声调语言仅提供基础元音标签。标准软最大化要么将两者视为无关类别，削弱跨语言共享；要么合并声调，丢失声调语言所需区分。本文提出潜空间软最大化（Latent Softmax），一种与连接时序分类（CTC）兼容的输出层，将带声调元音作为子类、基础元音作为主类，辅以单例的辅音和空白标签。当仅观测到基础元音主类标签时，带声调元音子类被视为潜在变量并被边缘化。在AISHELL-1中文和LibriSpeech英文数据集上的多语言实验表明，与标准软最大化基线相比，该方法在AISHELL-1上将语音到音素（S2P）音素错误率降低8.4%，在LibriSpeech test-clean上降低17.5%，test-other上降低12.6%。改进的语音到音素编码器也持续提升了大语言模型音素转字符（LLM-P2G）和投影器接口的词错误率表现。在中英混合语境下的码转换适配后，投影器接口在ASRU2019上将混合错误率降低2.6%，在CS-Dialogue上降低9.5%；而LLM-P2G结果未呈现一致优势。
原文摘要 · Abstract (English)
Phoneme-based multilingual automatic speech recognition (ASR) can share acoustic evidence across languages more directly than language-specific subword modeling. When tonal and non-tonal languages are jointly trained, however, their supervision granularity does not match: tonal languages annotate tone-marked vowels, whereas non-tonal languages typically provide only base-vowel labels. A standard softmax either treats the two as unrelated classes, weakening cross-lingual sharing, or collapses tones, losing distinctions required by tonal languages. We propose Latent Softmax, a connectionist temporal classification (CTC)-compatible output layer that models tone-marked vowels as subclasses and base vowels as major classes, while consonants and the CTC blank remain singleton labels. When only a base-vowel major-class label is observed, the tone-marked vowel subclass is treated as latent and marginalized out. Multilingual experiments on AISHELL-1 Mandarin and LibriSpeech English show that Latent Softmax reduces speech-to-phoneme (S2P) phoneme error rates over a standard softmax multilingual baseline by 8.4% on AISHELL-1, 17.5% on LibriSpeech test-clean, and 12.6% on test-other. The improved speech-to-phoneme encoders also yield consistent word error rate gains for both large-language-model phoneme-to-grapheme conversion (LLM-P2G) and projector-based interfaces. After code-switching adaptation in the evaluated Mandarin--English setting, Latent Softmax reduces projector-based mixed error rate by 2.6% on ASRU2019 and 9.5% on CS-Dialogue, whereas the LLM-P2G results do not establish a consistent advantage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。