arXiv:2603.14337cs.CV2026-03被引 2

发现大模型推理中注意力聚集的隐藏机制,提升多模态问答性能。

On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs

  • 分析注意力汇聚点(sink)在多模态模型中的作用机制。
  • 提出OutRo方法,通过特征对齐和提前解码优化推理效果。
  • 无需额外训练,在7个视频问答数据集上均表现更优。

本文旨在不增加训练的前提下,增强多模态大语言模型(Omni-LLMs)的推理能力。这类模型联合处理视频、音频和文本,因输入 token 数量庞大,注意力分配方式直接影响其行为。研究聚焦于注意力汇聚点(attention sinks),即无论语义内容如何都吸收大量注意力的特殊 token。通过系统分析,发现:(i) 高注意力汇聚并非仅反映头冗余,表明汇聚点的值向量具有额外功能;(ii) 汇聚点值向量作为全局偏置,叠加至每个 token 输出,统摄整体表征结构。基于此,提出 OutRo:将非汇聚点表示与汇聚点在特征空间对齐,并在早期层放宽汇聚点的因果掩码,以强化该全局信号。该设计无需额外前向传播或注意力图访问,实验表明,OutRo 在七个视频问答基准上持续提升性能,具备强泛化能力,且推理开销仅增加1.1倍。

原文摘要 · Abstract (English)

The goal of this paper is to strengthen the reasoning of Omnimodal Large Language Models (Omni-LLMs) at inference time, without additional training. These models jointly process video, audio, and text, and given the large number of tokens they consume, how attention is routed across them is central to their behaviour. We focus specifically on attention sinks, tokens that absorb a disproportionate share of attention mass regardless of their semantic content, to understand how this routing unfolds. To this end, we conduct a systematic analysis of sink behaviour in Omni-LLMs. Our analysis yields two key findings: (i) high sink attention does not solely indicate head redundancy, suggesting that sink value representations play additional functional roles; (ii) the sink value vector acts as a shared bias added to every token's output, serving as a global signal that organises the representation as a whole. Building on this, we propose OutRo, which correspondingly aligns non-sink token representations with the sink in feature space, and relaxes the causal mask for sink tokens at an early layer to sharpen this bias before the rest of decoding proceeds. This design enhances the reasoning process without requiring additional forward passes or access to attention maps. Based on extensive experiments, OutRo consistently improves performance on seven video QA benchmarks and demonstrates strong generalisation, while incurring only a 1.1x decoding overhead.

多模态注意力机制推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。