arXiv:2606.18273cs.CLcs.AI2026-06

让音频大模型在生成回答前先‘思考’声音细节,提升理解力。

Continuous Audio Thinking for Large Audio Language Models

论文配图:Continuous Audio Thinking for Large Audio Language Models
图 1 · 摘自论文原文
  • 用连续潜空间存留语音特征,通过专家知识蒸馏优化。
  • 在多个音频任务上显著提升性能,最高增益达6.2%。
  • 无需额外推理成本,适合部署于实时音频应用。

大型音频语言模型(LALMs)在语音识别、音乐分析等任务中表现优异,但因其训练目标为生成对齐文本,隐藏状态逐渐偏向文本生成,导致语音中的发音细节、语调、声音事件、情感和音高等丰富信息丢失,难以用于最终响应。为此,本文提出连续音频思考(CoAT)框架,通过从音频专家蒸馏知识,在生成响应前引入一个连续的潜在工作空间来组织声学信息。该思考空间可支持模型利用专家提供的丰富声学线索进行响应生成。此外,所提出的连续思考模块可在单次预填充阶段完成处理,不增加基线模型的自回归解码开销。在涵盖音频推理、音频理解、音乐分类、语音情绪识别和语音转录的广泛基准测试中,Qwen2-Audio、Qwen2.5-Omni-7B 和 Audio Flamingo~3 三个 LALM 均取得显著性能提升。进一步分析证实,辅助监督信号能从思考位置有效传播至文本输出。

原文摘要 · Abstract (English)

Large audio language models (LALMs) have shown impressive capabilities on diverse audio understanding tasks, ranging from speech transcription to music analysis. However, because LALMs are typically trained to produce text-aligned responses, their hidden states are progressively shaped for text generation rather than for preserving acoustic information. As a result, the diverse acoustic content that audio carries, such as phonetic detail, prosody, sound events, affect, and pitch, is lost along the way and difficult to leverage in the response. We introduce Continuous Audio Thinking (CoAT), a framework that equips audio language models with a continuous latent workspace for organizing acoustic information prior to response generation, grounded by distillation from audio experts. Within the thinking space, the model can utilize the rich acoustic information provided by expert distillation when generating its response. Furthermore, the proposed continuous thinking block can be processed in a single prefill, so CoAT does not require additional autoregressive decoding cost over the baseline. Across three LALMs, Qwen2-Audio, Qwen2.5-Omni-7B, and Audio Flamingo~3, performance gains on a broad benchmark suite spanning audio reasoning, audio understanding, music classification, speech emotion, and speech transcription demonstrate the effectiveness of CoAT. Further analysis confirms that the auxiliary supervision propagates from the thinking positions to the model's textual responses.

音频理解大模型思考机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。