发现Transformer层间注意力信号有普适分布,可大幅降低大模型重排延迟。
Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking
- 通过分析各层注意力信号,提出选择性提取策略提升效率。
- 0.6B模型性能超越主流生成式方法,8B模型媲美14B强化学习模型。
- 适用于需高效推理的复杂推理场景,尤其适合资源受限部署。
零样本文档重排中,基于大语言模型的方法已从点式转向列表式与集合式,以提升计算效率。尽管成效显著,这些方法仍依赖生成评分或输出概率,面临推理延迟高和结果不一致的瓶颈。最近提出的上下文重排(ICR)作为O(1)替代方案,直接提取内部注意力信号,避免文本生成开销。然而现有ICR方法仅简单聚合所有层信号,未探索层间贡献差异及跨架构一致性。此外,尚无统一研究在一致条件下对比生成、似然与内部注意力机制在多种重排框架中的表现。本文对生成、似然与内部注意力机制进行正交评估,发现注意力信号在各层呈现普遍的“钟形”分布,据此提出选择性ICR策略,使推理延迟降低30%-50%且不损失效果。在强调推理能力的BRIGHT基准测试中,精确捕捉高质量上下文注意力信号显著减少对模型规模与强化学习的依赖:零样本8B模型达到14B强化学习重排器性能,甚至0.6B模型也优于当前最优生成式方法。这些发现重新定义了基于大模型重排的效率-效果边界,凸显内部信号在复杂推理任务中的巨大潜力。代码与结果公开于https://github.com/ielab/Selective-ICR。
原文摘要 · Abstract (English)
Zero-shot document re-ranking with Large Language Models (LLMs) has evolved from Pointwise methods to Listwise and Setwise approaches that optimize computational efficiency. Despite their success, these methods predominantly rely on generative scoring or output logits, which face bottlenecks in inference latency and result consistency. In-Context Re-ranking (ICR) has recently been proposed as an O(1) alternative method. ICR extracts internal attention signals directly, avoiding the overhead of text generation. However, existing ICR methods simply aggregate signals across all layers; layer-wise contributions and their consistency across architectures have been left unexplored. Furthermore, no unified study has compared internal attention with traditional generative and likelihood-based mechanisms across diverse ranking frameworks under consistent conditions. In this paper, we conduct an orthogonal evaluation of generation, likelihood, and internal attention mechanisms across multiple ranking frameworks. We further identify a universal "bell-curve" distribution of relevance signals across transformer layers, which motivates the proposed Selective-ICR strategy that reduces inference latency by 30%-50% without compromising effectiveness. Finally, evaluation on the reasoning-intensive BRIGHT benchmark shows that precisely capturing high-quality in-context attention signals fundamentally reduces the need for model scaling and reinforcement learning: a zero-shot 8B model matches the performance of 14B reinforcement-learned re-rankers, while even a 0.6B model outperforms state-of-the-art generation-based approaches. These findings redefine the efficiency-effectiveness frontier for LLM-based re-ranking and highlight the latent potential of internal signals for complex reasoning ranking tasks. Our code and results are publicly available at https://github.com/ielab/Selective-ICR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。