大模型自训练会先升后崩，研究如何阻止这种自我崩溃。
Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training
- 通过多轮自训练实验发现模型在代码任务上快速提升后会突然失效。
- 7B模型经优化后最高通过率可达22.2%，但仍有近17%的性能下滑。
- 引入早停和奖励归一化等机制可缓解崩溃，但无法彻底消除风险。
自改进可能导致自退化。在代码任务的REINFORCE后训练中，模型在优化指标上迅速提升，随后在同一训练周期内崩溃。我们在受控的多种子测试环境中，使用Qwen-2.5-3B和Qwen-2.5-7B，在竞争性编程任务上进行10轮20步的训练，每轮采用二值CodeGrader奖励。结果表明，pass@1呈现明显的先升后降模式：在数十个梯度步骤内达到峰值后回落，有时降至接近零。这并非跨任务灾难性遗忘，而是固定分布下的策略过优化；KL与EWC类约束无法阻止该现象。我们探讨控制环应置于何处，对比了三种方案：CARE（跨轮次记忆机制，含能力后验、迁移门与回归感知信念更新）、ES（轮内早停规则，回滚至峰值检查点并增加3步预算）以及GRPO（使用组相对奖励归一化修改强化学习更新）。结果依赖于模型规模：在脆弱的Qwen-2.5-3B上，CARE v2将链末期的pass@1从4.9%提升至9.5%（95%置信区间[+0.4,+8.9]），5个种子中有4个获益；在Qwen-2.5-7B上，CARE达13.8%（与原版11.8%相当），而ES达22.2%（[14.1,28.0]）；出箱的GRPO达20.7%（[15.7,25.1]），接近REINFORCE+ES效果。尽管如此，其7B模型收益主要来自跨轮次信息传递，而轮内峰终差距仍维持约17个百分点。结合GRPO+ES时，3个种子中2个改善，但一个最终崩塌导致均值为17.0%（[0.0,28.1]）。对Gemma-3-4B的初步实验显示相同特征，表明该现象不限于Qwen。
原文摘要 · Abstract (English)
Self-improvement can self-regress. In REINFORCE post-training for code, a model can quickly improve on its optimized metric and then collapse within the same training campaign. We study this in a controlled multi-seed testbed using Qwen-2.5-3B and Qwen-2.5-7B, trained on competitive-programming tasks with binary CodeGrader reward across 10 sequential 20-step campaigns. Across campaigns, pass@1 shows a robust rise-then-collapse pattern: it peaks within tens of gradient steps and then falls back, sometimes to near zero. This is not cross-task catastrophic forgetting, but within-task policy over-optimization on a fixed distribution; KL- and EWC-style constraints do not prevent it. We ask where the control loop should sit. We compare three levels: CARE, a between-campaign memory mechanism with a capability posterior, transfer gate, and regression-aware belief revision; ES, a within-campaign early-stop rule that rolls forward the peak checkpoint and sets the next budget to peak_step+3; and GRPO, which changes the RL update using group-relative reward normalization. The answer is regime-dependent. On Qwen-2.5-3B, where naive REINFORCE is fragile, CARE v2 nearly doubles end-of-chain pass@1 from 4.9% to 9.5%, with paired bootstrap 95% CI [+0.4,+8.9] and gains in 4/5 seeds. On Qwen-2.5-7B, CARE reaches parity with naive REINFORCE, 13.8% vs. 11.8%, while ES reaches 22.2% [14.1,28.0]. Out-of-the-box GRPO reaches 20.7% [15.7,25.1], nearly matching REINFORCE+ES. GRPO raises the floor but does not remove the cliff. Its 7B gain mainly comes from better between-campaign carryover, while the within-campaign peak-to-end gap remains about 17 points under both REINFORCE and GRPO. GRPO+ES gives mixed evidence: 2/3 seeds improve, but one final cliff lowers the mean to 17.0% [0.0,28.1]. A Gemma-3-4B pilot shows the same signature, suggesting the phenomenon is not limited to Qwen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。