提出混合压缩方法,兼顾长文本局部细节与全局语义保留。
Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention
- 融合全局语义与局部重要性,动态决定保留或丢弃上下文。
- 在7个知识问答任务上平均提升13.1%性能,仅用11.2%的token量。
- 适合需要高效处理长文本的推理场景,尤其适合资源受限部署。
大语言模型在长序列推理中面临计算效率低下和冗余处理的问题,促使上下文压缩技术的发展。现有方法通常依赖于标记重要性进行硬式局部压缩,或通过隐变量编码实现软式全局压缩。然而,文本内容相关性的不均匀分布及用户指令需求的多样性常导致有价值信息丢失。为此,我们提出针对大语言模型的混合上下文压缩方法(HyCo₂),融合全局与局部视角,在保留关键语义的同时维护重要细节。具体地,利用混合适配器基于全局视图优化语义表达,不同适配器在不同任务上表现更优;引入分类层,根据局部视图为每个上下文标记分配保留概率,决定其去留。为促进全局与局部压缩的协同整合,我们在指令微调前加入辅助改写与补全预训练。实验表明,该方法显著提升长文本推理能力并降低标记消耗:在七个知识密集型问答基准上平均性能提升13.1%,且在保持未压缩方法性能的同时,将标记使用量减少88.8%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) encounter significant challenges in long-sequence inference due to computational inefficiency and redundant processing, driving interest in context compression techniques. Existing methods often rely on token importance to perform hard local compression or encode context into latent representations for soft global compression. However, the uneven distribution of textual content relevance and the diversity of demands for user instructions mean these approaches frequently lead to the loss of potentially valuable information. To address this, we propose $\textbf{Hy}$brid $\textbf{Co}$ntext $\textbf{Co}$mpression (HyCo$_2$) for LLMs, which integrates both global and local perspectives to guide context compression while retaining both the essential semantics and critical details for task completion. Specifically, we employ a hybrid adapter to refine global semantics with the global view, based on the observation that different adapters excel at different tasks. Then we incorporate a classification layer that assigns a retention probability to each context token based on the local view, determining whether it should be retained or discarded. To foster a balanced integration of global and local compression, we introduce auxiliary paraphrasing and completion pretraining before instruction tuning. This promotes a synergistic integration that emphasizes instruction-relevant information while preserving essential local details, ultimately balancing local and global information retention in context compression. Experiments show that our HyCo$_2$ method significantly enhances long-text reasoning while reducing token usage. It improves the performance of various LLM series by an average of 13.1\% across seven knowledge-intensive QA benchmarks. Moreover, HyCo$_2$ matches the performance of uncompressed methods while reducing token consumption by 88.8\%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。