用音素替代投影层，能提升语音识别性能，尤其在低资源场景下。
Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR
- 用音素序列直接输入大语言模型，取代传统特征投影层。
- 在英语和鞑靼语上，音素接口表现优于普通投影接口，低资源下优势更明显。
- 提出分组音素接口，兼顾效率与词边界信息，适合语音转写任务。
将预训练语音编码器与大语言模型（LLM）结合是语音识别的有前景方向，但性能与数据效率取决于语音-语言接口设计。常见做法是使用可学习的投影层将编码器特征映射到LLM嵌入空间，另一种选择是直接将离散音素序列输入LLM。在相同编码器与LLM架构下，我们在高资源英语（LibriSpeech）和低资源鞑靼语上对比了音素接口与基础投影接口。此外，我们提出一种BPE-音素接口，通过合并高频局部音素模式来提升效率，同时保留显式的词边界线索以支持音素到字形转换。在LibriSpeech上，音素接口性能与基础投影接口相当，而BPE-音素接口进一步提升效果；在鞑靼语上，音素接口显著优于基础投影接口。我们还发现音素监督可构建更强的音素感知混合接口，优于基础投影接口。
原文摘要 · Abstract (English)
Integrating pretrained speech encoders with large language models (LLMs) is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。