对比了高效微调与全量微调的理论差异，揭示其性能瓶颈。
Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning
- 从优化理论出发，证明高效微调是全量微调的严格子集。
- 实验验证在15个数据集上，高效微调在复杂任务中表现逊于全量微调。
- 适合关注模型表达能力与鲁棒性的研究者参考。
参数高效微调（PEFT）在显著降低计算资源消耗的同时，达到与全量微调（FFT）相当的性能，成为研究主流。然而我们发现，尽管在部分基准上表现良好，PEFT在复杂任务（如推理和指令微调）中仍落后于FFT。本文基于优化理论，从表征能力和鲁棒性角度比较两者特性。理论上证明了PEFT是FFT的严格子集，并给出PEFT的理论上限，表明受限的参数空间限制了模型的表征能力，使其更易受扰动影响。在涵盖分类、生成、推理、指令微调任务的15个数据集及11个对抗测试集上的实验验证了上述理论。我们希望这些结果能推动研究超越当前已成熟的PEFT范式。源代码见匿名GitHub仓库。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) methods achieve performance comparable to Full Fine-Tuning (FFT) while requiring significantly fewer computing resources, making it the go-to choice for researchers. We find that although PEFT can achieve competitive results on some benchmarks, its performance falls short of FFT in complex tasks, such as reasoning and instruction-based fine-tuning. In this paper, we compare the characteristics of PEFT and FFT in terms of representational capacity and robustness based on optimization theory. We theoretically demonstrate that PEFT is a strict subset of FFT. By providing theoretical upper bounds for PEFT, we show that the limited parameter space constrains the model's representational ability, making it more susceptible to perturbations. Experiments on 15 datasets encompassing classification, generation, reasoning, instruction fine-tuning tasks and 11 adversarial test sets validate our theories. We hope that these results spark further research beyond the realms of well established PEFT. The source code is in the anonymous Github repository\footnote{https://github.com/misonsky/PEFTEval}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。