用注意力分数重排长文本,无需人工标注就能精准匹配查询。
Query-focused and Memory-aware Reranker for Long Context Processing
- 基于模型注意力得分构建列表式重排框架,全局利用候选段落信息。
- 30亿参数小模型在多领域超越现有最优方法,LoCoMo基准创纪录。
- 可灵活扩展:加上下文信息或中层注意力头,效果更优且高效。
基于大语言模型中检索头的分析,我们提出一种替代性重排框架,通过选定注意力头的得分来估计段落与查询的相关性。该方法提供列表式解决方案,能在重排时利用整个候选短列的全局信息。同时自然生成连续相关性分数,支持在任意检索数据集上训练,无需李克特量表标注。框架轻量高效,仅需30亿参数的小模型即可达到优异性能。大量实验表明,该方法在多个领域(包括维基百科和长叙事数据集)均优于现有最优点对点和列表式重排器。在评估对话理解与记忆使用的LoCoMo基准上建立新纪录。进一步证明框架具备灵活性:通过为候选段落增加上下文信息可提升准确率,从中间层训练注意力头可在不牺牲性能的前提下提高效率。
原文摘要 · Abstract (English)
Built upon the existing analysis of retrieval heads in large language models, we propose an alternative reranking framework that trains models to estimate passage-query relevance using the attention scores of selected heads. This approach provides a listwise solution that leverages the holistic information within the entire candidate shortlist during ranking. At the same time, it naturally produces continuous relevance scores, enabling training on arbitrary retrieval datasets without requiring Likert-scale supervision. Our framework is lightweight and effective, requiring only small-scale models, such as 3B parameters, to achieve strong performance. Extensive experiments demonstrate that our method outperforms existing state-of-the-art pointwise and listwise rerankers across multiple domains, including Wikipedia and long narrative datasets. It further establishes a new state-of-the-art on the LoCoMo benchmark, which assesses dialogue understanding and memory usage. We further demonstrate that our framework supports flexible extensions. For example, augmenting candidate passages with contextual information further improves ranking accuracy, while training attention heads from middle layers enhances efficiency without sacrificing performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。