通过熵平衡提升检索增强生成的长上下文适应能力
BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation
- 基于熵不变性设计注意力机制，分离敏感度与上下文长度
- 在多个任务上实现比基线更高的准确率，支持零样本多重要性估计
- 适合需要长上下文处理的复杂问答与知识推理场景
随着大语言模型的快速发展，检索增强生成（RAG）已成为弥补其固有知识局限性的关键方法。然而，由于检索信息量通常较大，RAG常面临长上下文问题。从熵工程视角看，无约束的熵增长和注意力稀释是影响性能的关键因素。本文提出平衡熵工程的RAG（BEE-RAG）框架，通过熵不变性原则提升RAG对不同上下文长度的适应性。利用平衡上下文熵重构注意力动态，使注意力敏感度与上下文长度解耦，维持稳定的熵水平。在此基础上，引入零样本多重要性估计策略及参数高效自适应微调机制，以获取不同场景下的最优平衡因子。在多个RAG任务上的广泛实验验证了BEE-RAG的有效性。
原文摘要 · Abstract (English)
With the rapid advancement of large language models (LLMs), retrieval-augmented generation (RAG) has emerged as a critical approach to supplement the inherent knowledge limitations of LLMs. However, due to the typically large volume of retrieved information, RAG tends to operate with long context lengths. From the perspective of entropy engineering, we identify unconstrained entropy growth and attention dilution due to long retrieval context as significant factors affecting RAG performance. In this paper, we propose the balanced entropy-engineered RAG (BEE-RAG) framework, which improves the adaptability of RAG systems to varying context lengths through the principle of entropy invariance. By leveraging balanced context entropy to reformulate attention dynamics, BEE-RAG separates attention sensitivity from context length, ensuring a stable entropy level. Building upon this, we introduce a zero-shot inference strategy for multi-importance estimation and a parameter-efficient adaptive fine-tuning mechanism to obtain the optimal balancing factor for different settings. Extensive experiments across multiple RAG tasks demonstrate the effectiveness of BEE-RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。