通过重加权注意力,解决检索中关键词偏倚和注意力集中问题。
ReAttn: Improving Attention-based Re-ranking via Attention Re-weighting
- 对注意力权重进行跨文档逆文档频率加权,降低重复词影响。
- 引入熵正则化使注意力分布更均衡,避免集中在少数词上。
- 无需训练或标注,可直接提升现有注意力重排方法效果。
近期大型语言模型在零样本重排任务中表现出色。基于注意力的重排方法通过直接利用注意力权重生成相关性分数,具有高效且可解释的优点。然而,仍存在两大缺陷:一是注意力信号高度集中在少数文档的少量词上,导致其他词无法区分;二是注意力过度强调与查询词汇相似的短语,造成偏倚,使仅在词汇层面相似的无关文档被误判为相关。本文提出ReAttn,一种后处理注意力重加权策略。首先计算跨文档逆文档频率(IDF)权重,降低在候选文档中频繁出现的查询重叠词的注意力,减少词汇偏倚,突出独特信息词;随后采用基于熵的正则化,缓解注意力过度集中问题,促进信息词间的均衡分布。两项调整均直接作用于已有注意力权重,无需额外训练或监督。大量实验验证了该方法的有效性。
原文摘要 · Abstract (English)
The strong capabilities of recent Large Language Models (LLMs) have made them highly effective for zero-shot re-ranking task. Attention-based re-ranking methods, which derive relevance scores directly from attention weights, offer an efficient and interpretable alternative to generation-based re-ranking methods. However, they still face two major limitations. First, attention signals are highly concentrated a small subset of tokens within a few documents, making others indistinguishable. Second, attention often overemphasizes phrases lexically similar to the query, yielding biased rankings that irrelevant documents with mere lexical resemblance are regarded as relevant. In this paper, we propose \textbf{ReAttn}, a post-hoc re-weighting strategy for attention-based re-ranking methods. It first compute the cross-document IDF weighting to down-weight attention on query-overlapping tokens that frequently appear across the candidate documents, reducing lexical bias and emphasizing distinctive terms. It then employs entropy-based regularization to mitigate over-concentrated attention, encouraging a more balanced distribution across informative tokens. Both adjustments operate directly on existing attention weights without additional training or supervision. Extensive experiments demonstrate the effectiveness of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。