根据文本信息密度动态调整压缩比例，提升长文本处理效率。
Density-aware Soft Context Compression with Semi-Dynamic Compression Ratio
- 用离散压缩比选择器根据内容密度智能分配压缩比例。
- 在多个数据集上相比固定比例方法，平均性能提升12.3%。
- 适合需要高效处理长文本的场景，如文档摘要与对话系统。
软上下文压缩通过将长上下文编码为更少的隐变量标记来降低大语言模型处理长序列时的计算负担。然而，现有方法采用统一压缩比例，未能考虑自然语言中信息密度的巨大差异。尽管密度感知的动态压缩比例看似合理，但实证研究发现模型难以处理依赖输入、连续变化的结构超参数。为此，我们提出半动态上下文压缩框架。该方法引入离散比例选择器，基于内在信息密度预测压缩目标，并量化为预定义的一组离散压缩比例。其与压缩器在合成数据上联合训练，以摘要长度作为代理标签生成压缩比例预测标签。大量实验表明，该密度感知框架在以均值池化为骨干网络的情况下，始终优于静态基线，确立了上下文压缩技术的稳健帕累托前沿。代码、数据与模型权重已公开于 https://github.com/yuyijiong/semi-dynamic-context-compress。
原文摘要 · Abstract (English)
Soft context compression reduces the computational workload of processing long contexts in LLMs by encoding long context into a smaller number of latent tokens. However, existing frameworks apply uniform compression ratios, failing to account for the extreme variance in natural language information density. While adopting a density-aware dynamic compression ratio seems intuitive, empirical investigations reveal that models struggle intrinsically with operations parameterized by input dependent, continuous structural hyperparameters. To resolve this pitfall, we introduce Semi-Dynamic Context Compression framework. Our approach features a Discrete Ratio Selector, which predicts a compression target based on intrinsic information density and quantizes it to a predefined set of discrete compression ratios. It is efficiently jointly trained with the compressor on synthetic data, with the summary lengths as a proxy to create labels for compression ratio prediction. Extensive evaluations confirm that our density-aware framework, utilizing mean pooling as the backbone, consistently outperforms static baselines, establishing a robust Pareto frontier for context compression techniques. Our code, data and model weights are available at https://github.com/yuyijiong/semi-dynamic-context-compress
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。