揭示LoRA记忆的数学规律，提升大模型知识更新效率。
How LoRA Remembers? A Parametric Memory Law for LLM Finetuning
- 提出参数记忆定律，量化低秩适配的存储能力。
- 发现预测概率>0.5即能实现精确复现，存在确定性相变点。
- 设计内存优化策略MemFT，动态分配训练资源，提升记忆精度。
大语言模型需持续学习以适应动态现实环境。尽管低秩适配（LoRA）被广泛用于知识更新，但现有研究多依赖下游定性评估，对参数化记忆的定量容量与内在机制仍缺乏探索。为此，本文将LoRA作为潜空间中的可控记忆探针，系统量化其精确参数记忆能力。提出参数记忆定律——损失下降量ΔL与有效参数数及序列长度之间存在稳健幂律关系。在词元层面，细粒度分析揭示确定性相变现象：贪婪解码下，预测概率p > 0.5构成完整复现的充分条件。基于此，提出阈值引导优化策略MemFT，动态将训练预算分配至低于阈值的词元。实证表明，MemFT可显著提升记忆保真度与效率。代码将开源于https://github.com/zjunlp/ParametricMemoryLaw。
原文摘要 · Abstract (English)
Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。