arXiv:2602.09789cs.LG2026-02

大模型压缩反致信息失真,小模型反而更忠实还原原文。

When Less is More: The LLM Scaling Paradox in Context Compression

  • 用不同规模模型压缩上下文,发现越大越容易篡改事实
  • 27组实验显示中等模型在还原准确率上表现最佳
  • 适合关注模型可信度与信息保真度的研究者

长期以来,扩大模型参数量被视为提升生成能力的通用范式。然而,在有损上下文压缩场景下,我们发现一个「规模-保真悖论」:压缩器规模越大,重建内容的忠实度反而下降,尽管重建误差降低。在覆盖多个模型家族、规模和压缩率的27组设置中,该现象由两大因素主导:1)知识覆盖——大模型逐渐用自身先验替代源事实,如将“白草莓”改为“红草莓”;2)语义漂移——大模型倾向于改写而非复现原文,如将“爱丽丝打了鲍勃”变为“鲍勃打了爱丽丝”。有趣的是,这一悖论在各类设置中均持续存在,中等规模压缩器往往比更大模型更擅长忠实恢复。通过嵌入几何与重建确定性分析,我们发现压缩器倾向于将记忆组织在更宽泛的语义子空间中,导致表征模糊,易引发覆盖、漂移与恢复能力弱化。这些发现补充了现有上下文压缩评估体系,揭示当目标从合理生成转向忠实保存时,扩展规律的失效。

原文摘要 · Abstract (English)

Scaling up model parameters has long been a prevalent training paradigm driven by the assumption that larger models yield superior generation capabilities. However, under lossy context compression in a compressor--decoder setup, we find a \textbf{\textit{Size-Fidelity Paradox}}: increasing compressor size can lessen the faithfulness of reconstructed contexts though reconstruction error decreases. Across 27 compressor setups spanning model families, scales, and compression rates, we coin this paradox arising from two dominant factors: 1) \textit{knowledge overwriting}: larger models increasingly replace source facts with their own prior beliefs, \textit{e.g.}, ``the white strawberry`` $\to$ ``the red strawberry``; and 2) \textit{semantic drift}: larger models tend to paraphrase or restructure content instead of reproducing it verbatim, \textit{e.g.}, ``Alice hit Bob`` $\to$ ``Bob hit Alice``. Interestingly, this paradox persists across varied settings, with mid-sized compressors often outperforming larger ones in faithful recovery. By analyzing the compressed memory via embedding geometry and reconstruction determinacy, we further reveal that compressors tend to organize memory across broader semantic subspaces, yielding more ambiguous representations prone to overwriting, drift, and weakened recovery. These findings complement existing evaluations of context compression and expose a breakdown of scaling laws when the objective shifts from plausible generation to faithful preservation.

大模型信息保真压缩悖论

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。