大模型越学越记，小模型越学越懂，容量决定学习方式。
Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers
- 用可控任务测试不同大小模型的泛化与记忆能力
- 小模型能算新算式但记不住事实，大模型反之
- 联合训练时所有模型都丧失泛化能力，适合小模型研究者参考
大型语言模型中记忆与泛化的关联仍是开放问题，现有证据表明二者深度交织。本文通过从头预训练一系列容量受限的Transformer模型，在两个设计好的合成字符级任务上分别考察泛化（通过算术外推）和记忆（通过事实回忆）能力。结果呈现一致权衡：小模型能外推未见算式但无法记忆事实，大模型则相反；中等容量模型也表现出向记忆倾斜的趋势。当模型联合训练两个任务时，无论规模大小，均无法实现外推。这些发现表明预训练可能内在偏好某一学习模式。本研究在受控环境下揭示了模型容量如何影响学习行为，对小型语言模型的设计与部署具有广泛启示。
原文摘要 · Abstract (English)
The relationship between memorization and generalization in large language models (LLMs) remains an open area of research, with growing evidence that the two are deeply intertwined. In this work, we investigate this relationship by pre-training a series of capacity-limited Transformer models from scratch on two synthetic character-level tasks designed to separately probe generalization (via arithmetic extrapolation) and memorization (via factual recall). We observe a consistent trade-off: small models extrapolate to unseen arithmetic cases but fail to memorize facts, while larger models memorize but fail to extrapolate. An intermediate-capacity model exhibits a similar shift toward memorization. When trained on both tasks jointly, no model (regardless of size) succeeds at extrapolation. These findings suggest that pre-training may intrinsically favor one learning mode over the other. By isolating these dynamics in a controlled setting, our study offers insight into how model capacity shapes learning behavior and offers broader implications for the design and deployment of small language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。