首个不依赖外部模型的单栈语音识别系统，用稀疏专家路由提升准确率。
Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR
- 单栈架构处理语音和文本，用模态感知稀疏专家路由区分语音与文本路径。
- 113M参数模型在LibriSpeech上错误率2.8%（对比139M基线3.2%），多语言通用语音库降错1.6个百分点。
- 无需预训练语言模型或对齐模块，激活参数更少，适合资源受限场景部署。
我们提出一种用于自动语音识别（ASR）的解码器仅有的Conformer模型，该模型在单一堆叠中处理语音和文本，无需外部语音编码器或预训练大语言模型（LLM）。模型采用模态感知的稀疏专家混合（MoE）：语音与文本使用独立的专家池，通过硬路由和顶1选择机制实现分离。嵌入在混合因果性Conformer块中（语音为双向，文本为因果）。训练结合语音位置的CTC损失与文本生成的标签平滑交叉熵。我们的11300万参数模型在LibriSpeech上持续优于13900万参数的AED基线：test-clean错误率为2.8%（对比3.2%），test-other为5.6%（对比6.0%）。在包含五种语言的Common Voice 16.1上，单个多语言模型平均错误率从12.2%降至10.6%。据我们所知，这是首个通过模态感知路由与稀疏MoE，以随机初始化方式超越强基线的解码器仅有的ASR模型，在更少活跃参数下实现更高准确率，且无需对齐或适配模块。
原文摘要 · Abstract (English)
We present a decoder-only Conformer for automatic speech recognition (ASR) that processes speech and text in a single stack without external speech encoders or pretrained large language models (LLM). The model uses a modality-aware sparse mixture of experts (MoE): disjoint expert pools for speech and text with hard routing and top-1 selection, embedded in hybrid-causality Conformer blocks (bidirectional for speech, causal for text). Training combines CTC on speech positions with label-smoothed cross-entropy for text generation. Our 113M-parameter model consistently improves WER over a 139M AED baseline on Librispeech (2.8% vs. 3.2% test-clean; 5.6% vs. 6.0% test-other). On Common Voice 16.1 with a single multilingual model across five languages, our approach reduces average WER from 12.2% to 10.6%. To our knowledge, this is the first randomly initialized decoder-only ASR that surpasses strong AED baselines via modality-aware routing and sparse MoE, achieving better accuracy with fewer active parameters and without alignment/adaptation modules.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。