通过记忆-压缩循环提升大模型泛化能力，效果显著。
Memorization-Compression Cycles Improve Generalization
- 将语言建模重构为最小化表示熵的约束优化问题，实现记忆与压缩的动态平衡。
- 在GPT-2预训练中使表示熵降低50%，交叉熵下降4.8%，外推任务泛化提升35%。
- 适配于抗遗忘场景，通过分离表征减少干扰，表现接近睡眠巩固机制。
我们从理论上证明，泛化能力不仅可通过数据规模提升，还可通过压缩内部表示实现。为此，提出信息瓶颈语言建模（IBLM）目标，将语言建模转化为在保证最优预测性能前提下最小化表示熵的约束优化问题。实证发现，大模型预训练过程中存在自发的记忆-压缩周期，表现为交叉熵与基于矩阵的表示熵（MBE）梯度方向正负交替。该模式与IBLM的预测-压缩权衡高度一致，也类似生物清醒学习与睡眠巩固的交替过程。受此启发，提出门控相变（GAPT）训练算法，自适应切换记忆与压缩阶段。在FineWeb数据集上对GPT-2进行预训练，GAPT使MBE降低50%，交叉熵下降4.8%；在算术乘法外推任务中，泛化能力提升35%；在模拟灾难性遗忘的场景中，通过压缩与分离表征，使表征分离度提升97%，类比睡眠巩固功能。
原文摘要 · Abstract (English)
We prove theoretically that generalization improves not only through data scaling but also by compressing internal representations. To operationalize this insight, we introduce the Information Bottleneck Language Modeling (IBLM) objective, which reframes language modeling as a constrained optimization problem: minimizing representation entropy subject to optimal prediction performance. Empirically, we observe an emergent memorization-compression cycle during LLM pretraining, evidenced by oscillation positive/negative gradient alignment between cross-entropy and Matrix-Based Entropy (MBE), a measure of representation entropy. This pattern closely mirrors the predictive-compressive trade-off prescribed by IBLM and also parallels the biological alternation between awake learning and sleep consolidation. Motivated by this observation, we propose Gated Phase Transition (GAPT), a training algorithm that adaptively switches between memorization and compression phases. When applied to GPT-2 pretraining on FineWeb dataset, GAPT reduces MBE by 50% and improves cross-entropy by 4.8%. GAPT improves OOD generalizatino by 35% in a pretraining task on arithmetic multiplication. In a setting designed to simulate catastrophic forgetting, GAPT reduces interference by compressing and separating representations, achieving a 97% improvement in separation - paralleling the functional role of sleep consolidation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。