调整位置编号提升大模型上下文压缩效果
Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
- 通过重排位置编号，让压缩令牌更贴近原文本位置
- 在跨领域问答上平均提升1.9点ROUGE-1 F1
- 适用于需要高效压缩的文本与多模态场景
使用特殊标记（如概括、记忆或压缩标记）来压缩上下文信息是大语言模型的常见做法。然而，现有方法常忽略位置编码固有的局部归纳偏置，导致压缩过程忽视整体上下文依赖关系。我们提出一种简单而有效的方法——增强位置布局（EPL），仅通过调整位置编号（即标识令牌位置的数值）来提升大模型的上下文压缩能力。EPL使上下文标记与其对应特殊标记之间的距离最小化，同时保持上下文标记、特殊标记及后续标记间的位置顺序。将EPL集成到表现最佳的上下文压缩模型中，在跨领域问答数据集上平均提升1.9点ROUGE-1 F1；扩展至多模态场景时，视觉压缩大模型平均准确率提升2.6点。
原文摘要 · Abstract (English)
Using special tokens (e.g., gist, memory, or compressed tokens) to compress context information is a common practice for large language models (LLMs). However, existing approaches often neglect that position encodings inherently induce local inductive biases in models, causing the compression process to ignore holistic contextual dependencies. We propose \textbf{Enhanced Position Layout (EPL)}, a simple yet effective method that improves the context compression capability of LLMs by only adjusting position IDs, the numerical identifiers that specify token positions. EPL minimizes the distance between context tokens and their corresponding special tokens and at the same time maintains the sequence order in position IDs between context tokens, special tokens, and the subsequent tokens. Integrating EPL into our best performing context compression model results in a 1.9 ROUGE-1 F1 improvement on out-of-domain question answering datasets on average. When extended to multimodal scenarios, EPL leads to an average accuracy gain of 2.6 points for vision compression LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。