针对大模型微调时权重衰减失效问题，提出选择性惩罚新方法。
Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models
- 按层选择性施加权重衰减，不同层灵活调整更新强度。
- 在多个视觉与语言基准上提升分布内泛化与分布外鲁棒性。
- 适合需要稳定微调大模型的科研与工程场景。
现代优化器如AdamW结合动量和自适应学习率，有助于在训练初期逃离局部极小值并探索广阔参数空间，这对从头训练有利。但在从强大预训练模型继续微调时，这种探索可能造成参数偏离初始状态过远，降低模型鲁棒性和泛化能力。同时，对所有参数施加强正则化又易导致欠拟合。本文提出一种新的权重衰减技术——选择性投影衰减（Selective Projection Decay, SPD），仅对部分层施加强惩罚，允许其他层自由更新。直观上，SPD会扩展损失下降一致的层的搜索空间，收缩不一致层的搜索空间。实验表明，使用SPD的Adam在多个主流视觉与语言基准上均取得更优的分布内泛化与分布外鲁棒性表现。代码已公开于https://github.com/GT-RIPL/Selective-Projection-Decay.git。
原文摘要 · Abstract (English)
Modern optimizers such as AdamW, equipped with momentum and adaptive learning rate, are designed to escape local minima and explore the vast parameter space. This exploration is beneficial for finding good loss basins when training from scratch. It is not necessarily ideal when resuming from a powerful foundation model because it can lead to large deviations from the pre-trained initialization and, consequently, worse robustness and generalization. At the same time, strong regularization on all parameters can lead to under-fitting. We hypothesize that selectively regularizing the parameter space is the key to fitting and retraining the pre-trained knowledge. This paper proposes a new weight decay technique, Selective Projection Decay (SPD), that selectively imposes a strong penalty on certain layers while allowing others to change freely. Intuitively, SPD expands and contracts the parameter search space for layers with consistent and inconsistent loss reduction, respectively. Experimentally, when equipped with SPD, Adam consistently provides better in-distribution generalization and out-of-distribution robustness performance on multiple popular vision and language benchmarks. Code available at~\url{https://github.com/GT-RIPL/Selective-Projection-Decay.git}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。