用有限温度的吉布斯初始化解决大模型微调与强化学习的优化不匹配问题。
GIFT: Reconciling Post-Training Objectives via Finite-Temperature Gibbs Initialization
- 将标准微调视为零温极限，改用有限温度能量势函数建模监督信号。
- 在强化学习前驱阶段保留探索空间，显著提升下游任务性能。
- 适合需要稳定后训练流程的大型推理模型研发者使用。
当前大型推理模型（LRMs）的后训练范式——监督微调（SFT）后接强化学习（RL）——存在固有的优化不匹配：SFT中的严格监督导致分布坍塌，耗尽后续RL所需的探索空间。本文重新构建SFT以协调后训练目标，提出有限温度吉布斯初始化（GIFT）。我们将标准SFT视为抑制基础先验的退化零温极限，而GIFT则将监督信号建模为有限温度下的能量势，建立分布桥梁，促进整个后训练流程的目标一致性。实验表明，相较于标准SFT及其他竞争基线，GIFT在作为强化学习初始化时表现显著更优，提供了一条数学上严谨的路径，以保持探索能力并对齐两个后训练阶段。代码已公开于https://github.com/zzy1127/GIFT。
原文摘要 · Abstract (English)
The prevailing post-training paradigm for Large Reasoning Models (LRMs) - Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) - suffers from an intrinsic optimization mismatch: the rigid supervision inherent in SFT induces distributional collapse, thereby exhausting the exploration space necessary for subsequent RL. In this paper, we reformulate SFT to reconcile post-training objectives and propose Gibbs Initialization with Finite Temperature (GIFT). We characterize standard SFT as a degenerate zero-temperature limit that suppresses base priors. Conversely, GIFT incorporates supervision as a finite-temperature energy potential, establishing a distributional bridge that promotes objective consistency throughout the post-training pipeline. Our experiments demonstrate that GIFT significantly outperforms standard SFT and other competitive baselines when utilized for RL initialization, providing a mathematically principled pathway to preserve exploration and align the two post-training stages. Our code is available at https://github.com/zzy1127/GIFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。