用稀疏记忆层微调，减少大模型学新任务时的遗忘问题。
Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
- 只更新读取最频繁的记忆行，降低对原模型的干扰。
- 在医学考试任务上提升2.5个百分点，遗忘度低于1点。
- 适合需要保持通用能力的轻量级模型定制场景。
将预训练语言模型适配到新任务时常导致原有通用能力下降，即灾难性遗忘。稀疏记忆微调（SMF）通过向模型添加键值记忆层，并在每轮训练中仅更新当前批次读取最频繁的小部分记忆行，以缓解此问题。我们在Qwen-2.5-0.5B-Instruct上重新实现SMF，与LoRA及全参数微调在MedMCQA（四选一医学考试任务）上进行对比，使用WikiText困惑度和TriviaQA准确率作为遗忘探测指标。SMF在MedMCQA上提升2.5个百分点，同时两项遗忘指标均保持在基线模型的约1点以内；而LoRA和全微调虽有更大提升，但遗忘现象明显。我们还比较了两种行选择规则（KL散度与TF-IDF），二者在两类遗忘指标间权衡不同。
原文摘要 · Abstract (English)
Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。