提出PSFT方法，让大模型微调后更稳定、泛化能力更强。
Proximal Supervised Fine-Tuning
- 基于信任域思想，限制微调时模型参数变化幅度
- 在数学与人类价值观任务上，泛化性能优于传统微调
- 适合需要长期优化和保持模型多样性的场景
基础模型的监督微调（SFT）常导致原有能力退化。受强化学习中信任域策略优化（TRPO）和近端策略优化（PPO）启发，我们提出近端监督微调（PSFT）。该方法将SFT视为具有恒定正优势的策略梯度方法，通过引入信任域机制有效控制微调过程中的策略漂移，稳定优化并提升泛化能力，同时为后续训练保留优化空间。在数学与人类价值观领域的实验表明，PSFT在领域内表现与传统SFT相当，但在跨域泛化上表现更优，长时间训练下仍保持稳定，未出现熵崩溃现象，且为后续优化提供了更强基础。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) of foundation models often leads to poor generalization, where prior capabilities deteriorate after tuning on new tasks or domains. Inspired by trust-region policy optimization (TRPO) and proximal policy optimization (PPO) in reinforcement learning (RL), we propose Proximal SFT (PSFT). This fine-tuning objective incorporates the benefits of trust-region, effectively constraining policy drift during SFT while maintaining competitive tuning. By viewing SFT as a special case of policy gradient methods with constant positive advantages, we derive PSFT that stabilizes optimization and leads to generalization, while leaving room for further optimization in subsequent post-training stages. Experiments across mathematical and human-value domains show that PSFT matches SFT in-domain, outperforms it in out-of-domain generalization, remains stable under prolonged training without causing entropy collapse, and provides a stronger foundation for the subsequent optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。