arXiv:2604.05074cs.CL2026-04ACL被引 1

通过调节参数控制语言模型的记忆强度,揭示记忆与泛化的关系。

Memory Dial: A Training Framework for Controllable Memorization in Language Models

  • 用单一参数α调节记忆压力,实现对记忆的精准控制。
  • α越大,已见样本准确率越高,未见样本表现稳定。
  • 大模型更易响应记忆调节,高频词更易被记住。

语言模型中的记忆现象虽被广泛研究,却难以分离和控制。理解模型何时、何地记忆对解释其预测至关重要,但现有方法均为事后分析:可检测训练后模型的记忆,却无法将记忆效应与架构、数据或优化过程解耦。本文提出Memory Dial训练框架,将记忆压力作为显式可调变量。该框架通过单个参数α,在标准交叉熵与温度锐化目标之间插值,生成一系列架构和训练设置相同的模型,仅记忆压力不同。在六种架构和五个基准上的实验表明:(1) α能可靠调控记忆压力,已见样本准确率随α单调上升,未见样本准确率保持稳定;(2) 大模型对记忆压力更敏感;(3) 频繁序列比稀有序列更易被记忆。额外分析显示该效应在多种锐化温度下稳健,与单温度交叉熵存在质的区别,可迁移至多语言场景,并能在自然出现的单次序列中被检测到。Memory Dial为研究记忆行为如何产生及与泛化交互提供了可控实验框架。

原文摘要 · Abstract (English)

Memorization in language models is widely studied but remains difficult to isolate and control. Understanding when and what models memorize is essential for explaining their predictions, yet existing approaches are post-hoc: they can detect memorization in trained models, but cannot disentangle its effects from architecture, data, or optimization. We introduce Memory Dial, a training framework that makes memorization pressure an explicit, controllable variable. Memory Dial interpolates between standard cross-entropy and a temperature-sharpened objective via a single parameter $α$, producing a family of models identical in architecture and training setup (within each sweep), differing only in memorization pressure. Experiments across six architectures and five benchmarks demonstrate that: (1) $α$ reliably controls memorization pressure, with seen-example accuracy increasing monotonically while unseen accuracy remains stable; (2) larger models are more responsive to memorization pressure; and (3) frequent sequences are easier to memorize than rare ones. Additional analyses show that the effect is robust across a range of sharpening temperatures, differs qualitatively from single-temperature cross-entropy, transfers to multilingual settings, and is detectable even on naturally occurring single-occurrence sequences. Memory Dial provides a controlled experimental framework for studying how memorization behavior emerges and interacts with generalization in language models.

记忆控制语言模型训练框架可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。