用编码器压缩长文本，让大模型高效处理长上下文。
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning
- 将长文本分块压缩为软提示，通过适配器接入解码器模型。
- 在长文本问答和摘要任务上优于8个顶尖方法，长文本基准领先。
- 兼容预训练模型，兼顾性能与计算效率，适合实际部署。
长上下文处理对大语言模型在多轮对话、代码生成和文档摘要等任务中日益重要。本文针对高长上下文性能、低计算复杂度与预训练模型兼容性之间的矛盾——即“不可能三角”——提出E2LLM（编码器延展大语言模型）。E2LLM将长文本分块，利用预训练文本编码器将其压缩为软提示，并通过适配器对齐解码器仅有的大语言模型。为提升模型基于软提示的推理能力，采用编码器输出重建和长上下文指令微调两种训练目标。大量实验表明，E2LLM在文档摘要和问答任务中，不仅效果和效率均超越8种先进方法，且在同等规模模型中于LongBench v2上表现最佳。
原文摘要 · Abstract (English)
Processing long contexts is increasingly important for Large Language Models (LLMs) in tasks like multi-turn dialogues, code generation, and document summarization. This paper addresses the challenges of achieving high long-context performance, low computational complexity, and compatibility with pretrained models -- collectively termed the ``impossible triangle''. We introduce E2LLM (Encoder Elongated Large Language Models), a novel approach that effectively navigates this paradox. E2LLM divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. To enhance the LLM's reasoning with these soft prompts, we employ two training objectives: encoder output reconstruction and long-context instruction fine-tuning. Extensive experiments reveal that E2LLM not only outperforms 8 state-of-the-art (SOTA) methods in effectiveness and efficiency for document summarization and question answering, but also achieves the best performance on LongBench v2 among models of comparable size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。