用注意力反应度筛选长文本，提速30%且效果不降。
You Only Use Reactive Attention Slice For Long Context Retrieval
- 基于注意力反应得分排序句子，动态选取相关片段。
- 在六项评测中接近截断中间法的准确率，但推理速度提升30%。
- 适合需要高效处理长上下文的部署场景，如问答系统。
为支持大语言模型（LLM）更长的上下文，本文提出一种基于注意力的检索方法——你只需使用反应性注意力切片（YOURA）。该方法通过反应得分衡量每个句子对查询的注意力响应强度，贪心地选取最相关的句子。YOURA生成整个输入上下文的词元级反应向量，并设计无嵌入依赖的句子提取算法EASY，实现句子到词元向量的映射。在三个开源预训练模型上，针对六个LongBench QA数据集进行评估，YOURA在保持与简单有效的截断中间法几乎相同质量的前提下，使vLLM推理吞吐量最高提升30%。
原文摘要 · Abstract (English)
Supporting longer context for Large Language Models (LLM) is a promising direction to advance LLMs. As training a model for a longer context window is computationally expensive, many alternative solutions, such as Retrieval Augmented Generation (RAG), have been used. However, most existing RAG methods adopt embedding-based retrieval that falls short on long contexts. To address such challenges, we propose an attention-based retrieval technique, You Only Use Reactive Attention slice (YOURA). YOURA leverages a novel retrieval heuristic called reaction score to rank the relevance of each sentence in the input context with the query sentence. Intuitively, we measure how the per-token attention score "reacts" to the query and greedily retrieves the most reactive sentences. Internally, YOURA generates a token-indexed vector (called reaction vector) for the whole input context. To map each sentence to the token-indexed vector, we propose an Embedding-Agnostic Sentence Yield (EASY), a best-effort token wiggling algorithm. We evaluate our retrieval technique on three open-source pre-trained LLM models across six LongBench QA datasets. Our technique achieves up to 30% vLLM inference throughput improvement for serving long-context queries with a nearly identical quality score to the simple yet effective truncate-middle approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。