SFT过训练会导致奖励排序倒置，熵坍缩是关键原因。
SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR
- 通过分析预训练后熵变化，识别高风险SFT检查点
- Qwen模型在深度增加时GRPO性能从0.806降至0.481
- 适用于大模型微调中避免失败的早期诊断
标准做法选择最高pass@1的SFT检查点在SFT压缩回滚分布时可能失效。对于二值奖励，组内优势方差期望为 $p(1{-}p)(g{-}1)/g$；当早期GRPO使 $p$ 低于 $p^*(g)$，多数组奖励相同，无法提供相对信号。研究了Qwen2.5-Coder-3B和DeepSeek-Coder-6.7B的SFT深度阶梯。Qwen在五种深度、三组种子下测试，预训练pass@1随深度上升，但峰值GRPO pass@10从0.806降至0.481（三种子均值，n=20）；预训练熵与GRPO结果正相关（ρ=+0.69）。DeepSeek中pass@1始终高于$p^*(8)=0.083$，GRPO结果仅压缩未反转。两阶段诊断法结合预训练熵筛选与早期GRPO熵监控，可提前标记高风险检查点并终止失败训练。简单KL正则化和标签平滑无法挽救此场景下的坍缩检查点，表明问题非普通超参误差。
原文摘要 · Abstract (English)
The standard heuristic of selecting the SFT checkpoint with the highest pass@1 for GRPO can fail when SFT compresses the rollout distribution. For binary rewards, the expected within group advantage variance is $p(1{-}p)(g{-}1)/g$; when early GRPO drives $p$ below $p^*(g)$, most groups have identical rewards and provide no group relative signal. We study SFT depth ladders for Qwen2.5-Coder-3B and DeepSeek-Coder-6.7B. We test Qwen2.5-Coder-3B across five depths and three seeds, and DeepSeek-Coder-6.7B across four matched depths and three seeds. On Qwen, pre RL pass@1 rises with SFT depth, but peak GRPO pass@10 falls from $0.806$ to $0.481$ (3 seed mean, $n{=}20$); pre RL entropy is positively associated with the GRPO outcome ($ρ{=}{+}0.69$). On DeepSeek, pass@1 remains far above $p^*(8){=}0.083$, and GRPO outcomes compress rather than invert. A two stage diagnostic, combining pre RL entropy triage with an early GRPO entropy monitor, flags high risk checkpoints and can stop failing runs early. Simple KL to reference regularisation and label smoothing variants do not rescue the collapsed Qwen checkpoint in our setting, suggesting the failure is not a trivial GRPO hyperparameter artefact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。