用状态空间模型替代Transformer，实现更轻量的音频语言模型。
State-Space Large Audio Language Models
- 用状态空间模型替换音频感知和语言模型中的Transformer结构。
- 参数量显著减少，闭合任务表现仍媲美传统Transformer模型。
- 适合资源受限场景，如移动端或实时语音处理应用。
大型音频语言模型（LALM）结合音频感知模型与大语言模型（LLM），展现出对输入音频进行推理、理解语义和意图的强大能力。然而，这类系统依赖于Transformer架构，其计算复杂度随序列长度呈二次增长，难以在内存和时间受限的场景中部署。近期，状态空间模型（SSMs）作为Transformer的替代方案出现。尽管已有研究成功将基于Transformer的音频感知模块替换为状态空间模型，但基于状态空间的LALM尚未被探索。本文首次提出将音频感知模块和大语言模型均替换为状态空间模型，构建首个基于状态空间的LALM。实验表明，该模型虽然参数量显著降低，但在多种数据集上的闭合任务表现仍可与基于Transformer的LALM相媲美。
原文摘要 · Abstract (English)
Large Audio Language Models (LALM) combine the audio perception models and the Large Language Models (LLM) and show a remarkable ability to reason about the input audio, infer the meaning, and understand the intent. However, these systems rely on Transformers which scale quadratically with the input sequence lengths which poses computational challenges in deploying these systems in memory and time-constrained scenarios. Recently, the state-space models (SSMs) have emerged as an alternative to transformer networks. While there have been successful attempts to replace transformer-based audio perception models with state-space ones, state-space-based LALMs remain unexplored. First, we begin by replacing the transformer-based audio perception module and then replace the transformer-based LLM and propose the first state-space-based LALM. Experimental results demonstrate that space-based LALM despite having a significantly lower number of parameters performs competitively with transformer-based LALMs on close-ended tasks on a variety of datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。