大模型能从死记硬背的数据中学会泛化，突破传统认知。
Rote Learning Considered Useful: Generalizing over Memorized Data in LLMs
- 先用无意义符号死记事实关系，再通过有意义提示学习泛化。
- 8个大模型实验显示，记忆数据可被重新解释并形成语义结构。
- 适合研究知识注入与模型安全的学者，也警示滥用风险。
死记硬背是一种基于重复的记忆方法。许多研究者认为，死记硬背会阻碍泛化，因其鼓励逐字记忆而非深层理解。这一担忧甚至延伸至需一定记忆的事实知识。本文挑战这一观点，证明大语言模型（LLMs）实际上可以对死记硬背的数据实现泛化。我们提出一种两阶段“记忆-泛化”框架：模型首先使用合成的语义无关关键标记，死记事实性的主谓关系；随后在少量语义有意义的提示上微调，学习泛化能力。8个大模型的实验证明，模型可通过语义提示重新解读记忆内容，表现为关键标记与语义提示之间涌现出结构化的、语义对齐的潜在表征。这一意外发现为高效知识注入开辟了新路径，同时也揭示了记忆数据被恶意重用的潜在风险。
原文摘要 · Abstract (English)
Rote learning is a memorization technique based on repetition. Many researchers argue that rote learning hinders generalization because it encourages verbatim memorization rather than deeper understanding. This concern extends even to factual knowledge, which inevitably requires a certain degree of memorization. In this work, we challenge this view and demonstrate that large language models (LLMs) can, in fact, generalize over rote memorized data. We introduce a two-phase "memorize-then-generalize" framework, where the model first rote memorizes factual subject-object associations using a synthetic semantically meaningless key token and then learns to generalize by fine-tuning on a small set of semantically meaningful prompts. Extensive experiments over 8 LLMs show that the models can reinterpret rote memorized data through the semantically meaningful prompts, as evidenced by the emergence of structured, semantically aligned latent representations between the key token and the semantically meaningful prompts. This surprising finding opens the door to both effective and efficient knowledge injection as well as possible risks of repurposing the memorized data for malicious usage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。