通过保持旋转不变性提升大模型微调的泛化能力
Rotation-Preserving Supervised Fine-Tuning
- 用投影旋转不变性代替昂贵的敏感方向计算
- 在数学推理任务上同时提升域内与域外性能
- 适合需要强泛化的下游微调场景
监督微调虽能提升域内表现，但会损害域外泛化能力。已有研究指出这与预训练权重矩阵主奇异子空间的变化有关，但直接通过海塞或费舍尔信息识别敏感方向在大模型规模下成本过高。本文提出旋转保持型监督微调（RPSFT），以保留预训练奇异子空间的投影旋转作为费舍尔敏感方向的高效代理。RPSFT对每个预训练权重矩阵的前k个奇异向量块的投影旋转进行惩罚，限制不必要的旋转同时保留任务适应性。在多个模型家族和规模上，基于数学推理数据训练的结果显示，RPSFT在域内/域外权衡上优于标准SFT及强基线方法，更好保留预训练表征，并为下游强化学习微调提供更优初始化。代码已开源。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) improves in-domain performance but can degrade out-of-domain (OOD) generalization. Prior work suggests that this degradation is related to changes in dominant singular subspaces of pretrained weight matrices. However, directly identifying loss-sensitive directions with Hessian or Fisher information is computationally expensive at LLM scale. In this work, we propose preserving projected rotations in pretrained singular subspaces as an efficient proxy for Fisher-sensitive directions, which we call Rotation-Preserving Supervised Fine-Tuning (RPSFT). RPSFT penalizes changes in the projected top-$k$ singular-vector block of each pretrained weight matrix, limiting unnecessary rotation while preserving task adaptation. Across model families and sizes trained on math reasoning data, RPSFT improves the in-domain/OOD trade-off over standard SFT and strong SFT baselines, better preserves pretrained representations, and provides stronger initializations for downstream RL fine-tuning. Code is available at \href{https://github.com/jinhangzhan/RPSFT.git}{https://github.com/jinhangzhan/RPSFT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。