研究如何用压缩令牌提升大模型长文本处理能力
A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compression
- 用核心信息令牌压缩上下文，替代完整注意力机制
- 在检索生成等任务中几乎无损，但合成回忆任务表现差
- 发现三类失效模式，提出两种改进策略
本文系统研究了基于核心信息的上下文压缩方法，以提升大语言模型处理长文本的能力。重点回答两个问题：（1）该方法能否替代全注意力模型？（2）压缩会引发哪些潜在失败模式？通过大量实验发现，该方法在检索增强生成和长文档问答任务中可实现近似无损性能，但在合成回忆任务中存在明显不足。我们识别出三种关键失败模式：边界丢失、意外内容丢失、途中信息丢失。为此，提出两种有效策略：细粒度自编码以增强原始标记重建，分段重要性估计以根据标记依赖调整优化。本工作为理解此类压缩机制提供了深刻见解，并提供了实用改进方案。
原文摘要 · Abstract (English)
In this work, we provide a thorough investigation of gist-based context compression methods to improve long-context processing in large language models. We focus on two key questions: (1) How well can these methods replace full attention models? and (2) What potential failure patterns arise due to compression? Through extensive experiments, we show that while gist-based compression can achieve near-lossless performance on tasks like retrieval-augmented generation and long-document QA, it faces challenges in tasks like synthetic recall. Furthermore, we identify three key failure patterns: lost by the boundary, lost if surprise, and lost along the way. To mitigate these issues, we propose two effective strategies: fine-grained autoencoding, which enhances the reconstruction of original token information, and segment-wise token importance estimation, which adjusts optimization based on token dependencies. Our work provides valuable insights into the understanding of gist token-based context compression and offers practical strategies for improving compression capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。