用专家路由让语音和文本各走专路，提升多模态理解能力。
MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
- 按输入类型路由到语音或文本专用专家，实现模态感知处理
- 在多个语音任务上表现超越同规模模型，尤其在语音问答中提升显著
- 全开源架构，适合需要可复现语音文本融合研究的团队
我们提出MoST（语音与文本混合），一种基于模态感知专家混合（MAMoE）架构的新型多模态大语言模型，能无缝融合语音与文本处理。现有模型对不同模态使用相同参数，忽略其表示差异；而我们设计了专用路由路径，根据输入类型将令牌导向适配的专家。MAMoE通过两类组件协同增强：模态专用专家组捕捉领域特异性模式，共享专家促进跨模态信息传递。基于此架构，我们构建了一条高效转换流程：在ASR与TTS数据集上进行策略性后训练，再在精心筛选的语音-文本指令数据集上微调。关键优势在于全程仅依赖完全公开的开源数据集，实现高性能与数据高效。在自动语音识别、语音合成、音频语言建模及语音问答等多个基准测试中，MoST始终优于同等参数量的现有模型。消融实验证实，模态专用路由机制与共享专家设计对所有测试领域的性能提升均有显著贡献。据我们所知，MoST是首个基于专家混合架构的全开源语音-文本大语言模型。模型、训练代码、推理代码及训练数据已公开于https://github.com/NUS-HPC-AI-Lab/MoST。
原文摘要 · Abstract (English)
We present MoST (Mixture of Speech and Text), a novel multimodal large language model that seamlessly integrates speech and text processing through our proposed Modality-Aware Mixture of Experts (MAMoE) architecture. While current multimodal models typically process diverse modality representations with identical parameters, disregarding their inherent representational differences, we introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type. MAMoE simultaneously enhances modality-specific learning and cross-modal understanding through two complementary components: modality-specific expert groups that capture domain-specific patterns and shared experts that facilitate information transfer between modalities. Building on this architecture, we develop an efficient transformation pipeline that adapts the pretrained MoE language model through strategic post-training on ASR and TTS datasets, followed by fine-tuning with a carefully curated speech-text instruction dataset. A key feature of this pipeline is that it relies exclusively on fully accessible, open-source datasets to achieve strong performance and data efficiency. Comprehensive evaluations across ASR, TTS, audio language modeling, and spoken question answering benchmarks show that MoST consistently outperforms existing models of comparable parameter counts. Our ablation studies confirm that the modality-specific routing mechanism and shared experts design significantly contribute to performance gains across all tested domains. To our knowledge, MoST represents the first fully open-source speech-text LLM built on a Mixture of Experts architecture. \footnote{We release MoST model, training code, inference code, and training data at https://github.com/NUS-HPC-AI-Lab/MoST
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。