用轻量级专家路由模块，让音频与语言模型高效对齐。
Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module
- 仅训练嵌入层内的专家路由模块，冻结音频编码器和语言模型。
- 在连续空间动态选择专家向量，实现音频表示到语言可理解空间的转化。
- 无需修改语言模型词汇，适合构建高效多模态智能体。
将预训练音频编码器与大语言模型（LLM）对齐，为构建强大的多模态智能体提供了一条参数高效的路径。然而，现有方法常需代价高昂的全模型微调，或依赖表达能力有限的静态适配器。受柏拉图表征假说启发，我们提出 SteerMoE——一种新颖且模块化的音频-语言对齐框架。该框架冻结音频编码器和LLM解码器，仅训练集成于编码器层中的轻量级引导模块。该模块利用混合专家（MoE）路由器动态选择并应用学习到的引导向量，逐步将连续音频表示转换为语言模型可理解的空间。由于完全在连续嵌入空间中操作，本方法无需修改LLM词表，同时保留其高级推理与代理能力。实验表明，SteerMoE在语音识别、音频理解及定性函数调用任务中均表现优异，兼具高度模块化与计算效率，为构建复杂音频-语言系统提供了稳健新范式。
原文摘要 · Abstract (English)
Aligning pretrained audio encoders and Large Language Models (LLMs) offers a promising, parameter-efficient path to building powerful multimodal agents. However, existing methods often require costly full-model finetuning or rely on static adapters that may lack expressive power. Drawing inspiration from the Platonic Representation Hypothesis, we introduce SteerMoE, a novel and modular framework for audio-language alignment. SteerMoE freezes both the audio encoder and the LLM decoder, training only a lightweight steering module integrated within the encoder's layers. This module uses a Mixture-of-Experts (MoE) router to dynamically select and apply learned steering vectors, progressively transforming continuous audio representations into a space comprehensible to the LLM. By operating entirely in the continuous embedding space, our approach requires no modifications to the LLM's vocabulary and preserves its advanced reasoning and agentic capabilities. We demonstrate through experiments on ASR, audio understanding, and a qualitative function-calling task that SteerMoE achieves strong performance while remaining highly modular and computationally efficient, offering a robust new paradigm for developing sophisticated audio-language systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。