低秩微调的记新能力受适配器容量限制，高秩更记新但易遗忘旧知识。
Learning New Facts with QLoRA: An Acquisition-Retention Frontier
- 用不同秩的QLoRA对比记新与保旧的平衡关系
- 高秩QLoRA记新效果更好，但对无关任务性能下降明显
- 适合关注参数高效微调中记忆-遗忘权衡的研究者
参数高效微调通常被认为能保留预训练能力，因为我们只更新少量参数。我们发现这一假设强烈依赖于适配器容量。在基于OpenStreetMap构建的受控基准上，让Qwen3-4B学习匿名地理关联的同时保留无关能力。对比全量微调（FFT）与量化低秩适配（QLoRA）在秩8、16、32、64下的表现，发现秩决定了明显的记新-保旧前沿：低秩QLoRA保持域外（OOD）性能好但记新少，高秩则提升同义改写泛化能力，代价是无关基准性能下降。FFT表现保守：保旧能力强，但无法达到最高记新水平。分布、权重空间与谱诊断均反映该权衡，高秩QLoRA远离预训练模型。独立数学适配实验显示前沿效应较弱，表明该现象最显著于需植入新事实而非强化已有技能时。代码与数据见https://github.com/zhngstl/new_facts_forgetting。
原文摘要 · Abstract (English)
Parameter-efficient fine-tuning is often assumed to preserve pretrained capabilities because it updates only a small number of parameters. We show that this assumption depends strongly on adapter capacity. We study factual acquisition in a controlled OpenStreetMap-derived benchmark where Qwen3-4B must acquire anonymized geographic associations while retaining unrelated capabilities. Comparing full fine-tuning (FFT) with quantized low-rank adaptation (QLoRA) at ranks 8, 16, 32, and 64, we find that rank induces a clear acquisition--retention frontier. Low-rank QLoRA preserves out-of-domain (OOD) performance but acquires fewer facts, whereas higher ranks improve same-fact paraphrase generalization at an increasing cost in performance on unrelated benchmarks. FFT behaves as a conservative baseline: it retains general capabilities well, but does not reach the highest factual-acquisition regime. Distributional, weight-space, and spectral diagnostics mirror this behavioral trade-off, with higher-rank QLoRA moving farther from the pretrained model. A separate math adaptation experiment shows a weaker frontier, suggesting that the effect is most pronounced when adaptation must install new factual associations rather than reinforce skills already supported by pretraining. Code and data are available at https://github.com/zhngstl/new_facts_forgetting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。