arXiv:2506.10423cs.SDcs.AI2025-06被引 4

提出轻量级音频融合方法,高效将音频语义注入大模型。

PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMs

  • 通过注意力机制在特定层注入音频表示,跳过前馈模块。
  • 相比现有方法提升30%性能,内存减少60%,吞吐量提高190%。
  • 适合需要高效音频-语言融合的实时应用开发者。

将音频感知融入大语言模型(LLMs)是实现机器听觉应用的新兴研究方向,但如何高效地从音频编码器传递丰富音频语义仍缺乏探索。当前主流方法将音频编码器输出映射到语言模型输入空间(如通过MLP或Q-Former),再插入文本序列中,我们称其为PLITS集成。本文提出一种轻量级替代方案LAL:仅通过选定层的注意力机制注入音频表示,绕过前馈模块,以适切抽象层级编码音频语义,显著降低计算开销。进一步提出混合方法PAL,对少量摘要令牌使用PLITS,而全音频序列通过LAL融合。在相同训练条件下,LAL在多个基础模型与任务上表现持平或超越现有方法,最高较强基线提升30%,内存减少约60%,吞吐量提升约190%。PAL则在保持性能的同时,显著优化计算与内存效率。

原文摘要 · Abstract (English)

Integration of audio perception into large language models (LLMs) is an emerging research area for enabling machine listening applications, yet efficient transfer of rich audio semantics from audio encoders to LLMs remains underexplored. The most widely used integration paradigm projects audio-encoder output tokens into the LLM input space (e.g., via an MLP or a Q-Former) and then prepends or inserts them into the text token sequence. We refer to this generic scheme as Prepend to the LLM's input token space (PLITS) integration. We propose an efficient alternative, Lightweight Audio LLM Integration (LAL). LAL injects audio representations solely through the attention mechanism at selected LLM layers, bypassing the feed-forward module. It encodes rich audio semantics at an appropriate level of abstraction for integration into different transformer blocks, substantially reducing computational overhead compared to existing approaches. We further introduce PAL, a hybrid integration approach for efficiently Probing Audio encoders via LLM. PAL applies PLITS only to a compact set of summary tokens while integrating the full audio token sequence via LAL. Under an identical training curriculum, LAL consistently matches or outperforms existing integration approaches across multiple base LLMs and tasks, with improvements of up to 30% over a strong PLITS baseline, while reducing memory usage by about 60% and increasing throughput by about 190%. Moreover, PAL matches or exceeds PLITS performance while offering substantially better computational and memory efficiency.

音频融合大模型轻量化注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。