提出一种压缩框架,让大模型高效处理长文本。
CCF: A Context Compression Framework for Efficient Long-Sequence Language Modeling
- 分段语义聚合+键值记忆编码,生成紧凑表征
- 高压缩比下保持低困惑度,吞吐量提升显著
- 适合需要长文本理解的场景,如文档分析
将语言模型扩展到更长上下文对捕捉长篇话语中的丰富依赖关系至关重要。然而,直接扩展上下文会带来显著的计算和内存开销,导致训练与推理效率低下。本文提出CCF,一种新型上下文压缩框架,通过学习层次化潜在表示,在大幅减少输入冗余的同时保留全局语义,实现高效的长序列建模。CCF结合分段语义聚合与键值记忆编码,形成紧凑表征,支持准确重构与长程理解。为进一步提升可扩展性,引入一种训练高效的优化策略,将增量分段解码与稀疏蓄水池采样相结合,显著降低内存开销且不损害性能。在多个长上下文语言建模基准上的实证结果表明,CCF在高压缩比下仍能取得具有竞争力的困惑度,并显著优于现有方法的吞吐量与内存效率。这些发现凸显了结构化压缩在可扩展、高效长上下文建模中的潜力。
原文摘要 · Abstract (English)
Scaling language models to longer contexts is essential for capturing rich dependencies across extended discourse. However, naïve context extension imposes significant computational and memory burdens, often resulting in inefficiencies during both training and inference. In this work, we propose CCF, a novel context compression framework designed to enable efficient long-context modeling by learning hierarchical latent representations that preserve global semantics while aggressively reducing input redundancy. CCF integrates segment-wise semantic aggregation with key-value memory encoding, forming compact representations that support accurate reconstruction and long-range understanding. To further enhance scalability, we introduce a training-efficient optimization strategy that couples incremental segment decoding with sparse reservoir sampling, substantially reducing memory overhead without degrading performance. Empirical results on multiple long-context language modeling benchmarks demonstrate that CCF achieves competitive perplexity under high compression ratios, and significantly improves throughput and memory efficiency compared to existing approaches. These findings highlight the potential of structured compression for scalable and effective long-context language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。