用注意力机制精准压缩检索文本，提升大模型生成效率
AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation
- 将查询转化为下一个词预测任务，聚焦语义核心词
- 在LongBench上实现6.3倍上下文压缩，性能比LLMLingua高10%
- 适合需要高效处理长文本的问答与生成系统
尽管RAG在大模型应用中表现出色，但不断增长的检索上下文长度带来了信息冗余和巨大计算开销。现有方法如LLMLingua缺乏上下文感知能力，压缩率控制灵活性差，常导致压缩不足或信息损失严重。本文提出AttentionRAG，一种基于注意力引导的RAG上下文压缩方法。其核心思想是通过注意力聚焦机制，将RAG查询重构为下一个词预测任务，将查询语义聚焦于单一关键词，从而实现查询与检索内容间精准高效的注意力计算。在LongBench和Babilong基准上的大量实验表明，AttentionRAG可实现最高6.3×的上下文压缩，且在关键指标上相比LLMLingua方法提升约10%。
原文摘要 · Abstract (English)
While RAG demonstrates remarkable capabilities in LLM applications, its effectiveness is hindered by the ever-increasing length of retrieved contexts, which introduces information redundancy and substantial computational overhead. Existing context pruning methods, such as LLMLingua, lack contextual awareness and offer limited flexibility in controlling compression rates, often resulting in either insufficient pruning or excessive information loss. In this paper, we propose AttentionRAG, an attention-guided context pruning method for RAG systems. The core idea of AttentionRAG lies in its attention focus mechanism, which reformulates RAG queries into a next-token prediction paradigm. This mechanism isolates the query's semantic focus to a single token, enabling precise and efficient attention calculation between queries and retrieved contexts. Extensive experiments on LongBench and Babilong benchmarks show that AttentionRAG achieves up to 6.3$\times$ context compression while outperforming LLMLingua methods by around 10\% in key metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。