探究LoRA秩大小对模型知识保留与泛化能力的影响
How Much is Too Much? Exploring LoRA Rank Trade-offs for Retaining Knowledge and Domain Robustness
- 系统测试不同秩值下的LoRA表现，对比全量微调
- 特定秩值下LoRA在推理任务上优于全量微调
- 揭示了注意力结构变化与表示漂移的内在机制
大语言模型越来越多地通过微调适配下游任务。全量监督微调（SFT）和参数高效微调（PEFT）方法（如低秩适应，LoRA）是两种主流方式。尽管PEFT因其计算效率被广泛应用，但其配置（如秩）对下游问答任务和泛化性能的影响仍缺乏深入研究。本文在多个推理与回忆数据集上进行系统评估，通过秩扫描量化SFT与PEFT之间的权衡。同时比较了PEFT与SFT在域内与域外适应中的准确率，揭示了不同的泛化行为和任务特异性遗忘现象。结果表明，特定秩值下LoRA在推理任务上可达到甚至超越SFT性能。此外，通过谱特征和层间注意力结构分析，揭示了表示漂移与注意力模式变化的内在机制。
原文摘要 · Abstract (English)
Large language models are increasingly adapted to downstream tasks through fine-tuning. Full supervised fine-tuning (SFT) and parameter-efficient fine-tuning (PEFT) methods, such as Low-Rank Adaptation (LoRA), are two dominant approaches. While PEFT methods are widely used for their computational efficiency, the implications of their configurations (e.g., rank) remain under-explored in downstream Q&A tasks and generalisation. In this work, we perform a comprehensive evaluation across multiple reasoning and recall datasets, conducting a rank sweep to quantify the trade-off between SFT and PEFT. We also compare the accuracy of PEFT and SFT models across in-domain and out-of-domain adaptation, highlighting distinct generalisation behaviour and task-specific forgetting. We demonstrate that LoRA achieves competitive and in some cases superior performance compared to SFT, particularly on reasoning tasks at specific rank values. Additionally, we analyze the internal representations via spectral features and layer-wise attention structures, offering insights into representational drift and structural changes in attention patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。