让大模型高效处理超长文本，按需压缩关键信息。
LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs
- 用循环压缩机制突破模型上下文长度限制，无需重训。
- 查询相关压缩使模型更精准保留有用信息，提升长文理解能力。
- 适合需要深度上下文分析的问答、摘要等任务。
尽管大型语言模型在生成连贯且上下文丰富的输出方面表现优异，但其处理长序列的能力受限于固定长度的位置嵌入。此外，处理长序列的计算成本随序列长度呈二次增长，难以扩展上下文长度。为此，我们提出长序列上下文循环压缩（LCIRC）方法，通过循环压缩实现对超出模型长度限制的长序列的高效处理，且无需重新训练整个模型。我们进一步引入查询相关上下文建模，选择性压缩与查询相关的部分，确保模型保留最相关的内容。实验结果表明，查询相关LCIRC（QD-LCIRC）显著提升了大模型处理扩展上下文的能力，适用于需要全面上下文理解与查询相关性的任务。
原文摘要 · Abstract (English)
While large language models (LLMs) excel in generating coherent and contextually rich outputs, their capacity to efficiently handle long-form contexts is limited by fixed-length position embeddings. Additionally, the computational cost of processing long sequences increases quadratically, making it challenging to extend context length. To address these challenges, we propose Long-form Context Injection with Recurrent Compression (LCIRC), a method that enables the efficient processing long-form sequences beyond the model's length limit through recurrent compression without retraining the entire model. We further introduce query dependent context modeling, which selectively compresses query-relevant information, ensuring that the model retains the most pertinent content. Our empirical results demonstrate that Query Dependent LCIRC (QD-LCIRC) significantly improves LLM's ability to manage extended contexts, making it well-suited for tasks that require both comprehensive context understanding and query relevance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。