数据分布影响大模型上下文压缩效果，研究发现输入熵与压缩质量负相关。
Data Distribution Matters: A Data-Centric Perspective on Context Compression for Large Language Model
- 从数据视角出发，分析输入与模型内生知识对压缩的影响
- 输入数据熵越低，压缩质量越高；模型内生差异会削弱压缩收益
- 提出优化压缩效果的实用建议，适合长文本处理研究者参考
大语言模型在长上下文场景中的部署受制于计算效率低下和信息冗余问题。尽管近期研究普遍采用上下文压缩来缓解这些问题，但现有工作仅关注模型侧改进，未探索数据分布本身对压缩质量的影响。本文首次从数据中心视角系统研究数据分布对压缩质量的影响，涵盖输入数据与模型内部预训练知识（内在数据）两个维度。通过基于自编码器的框架评估压缩表示的语义完整性，实验结果表明：(1) 编码器测量的输入熵与压缩质量呈负相关，而解码器测量的熵在固定解码器设置下无显著关联；(2) 编码器与解码器之间内在数据的差异显著降低压缩增益，且难以缓解。基于上述发现，本文进一步提出优化压缩性能的实用指南。
原文摘要 · Abstract (English)
The deployment of Large Language Models (LLMs) in long-context scenarios is hindered by computational inefficiency and significant information redundancy. Although recent advancements have widely adopted context compression to address these challenges, existing research only focus on model-side improvements, the impact of the data distribution itself on context compression remains largely unexplored. To bridge this gap, we are the first to adopt a data-centric perspective to systematically investigate how data distribution impacts compression quality, including two dimensions: input data and intrinsic data (i.e., the model's internal pretrained knowledge). We evaluate the semantic integrity of compressed representations using an autoencoder-based framework to systematically investigate it. Our experimental results reveal that: (1) encoder-measured input entropy negatively correlates with compression quality, while decoder-measured entropy shows no significant relationship under a frozen-decoder setting; and (2) the gap between intrinsic data of the encoder and decoder significantly diminishes compression gains, which is hard to mitigate. Based on these findings, we further present practical guidelines to optimize compression gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。