从稳定与可塑性视角评估高效微调，发现正交微调最优。
PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
- 提出新评测框架PEFT-Arena，兼顾任务性能与预训练能力保留。
- 正交微调在相同参数预算下表现最佳，达成更优稳定-可塑权衡。
- 发现微调后期易过拟合，可通过路径回溯优化效果。
参数高效微调（PEFT）已成为适配大语言模型的标准方法，但现有评估主要关注下游任务准确率，忽视了对预训练能力的保留。我们主张应从稳定-可塑性困境的角度评估PEFT：即目标任务适应性与遗忘抵抗之间的权衡。为此，我们提出PEFT-Arena基准，同时衡量下游性能与通用能力保留。在多种方法中，我们发现不同的稳定-可塑性特征；在相近参数预算下，正交微调达到最理想的帕累托前沿。为解释差异，我们从两个几何视角分析PEFT更新：在权重空间，谱分析揭示参数化如何与预训练奇异值结构相互作用；在激活空间，保留度量显示微调是否保持或扭曲通用能力表示，遗忘与非等距表示失真相关。最后分析表明，最终SFT检查点常超出更优的目标保留点。受此启发，我们展示了通过路径回溯实现后处理改进的案例。
原文摘要 · Abstract (English)
Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。