控制微调与强化学习数据重叠度,发现不重叠效果最佳且零成本提升性能。
SFT-GRPO Data Overlap as a Post-Training Hyperparameter for Autoformalization
- 通过调整SFT与GRPO训练数据重叠率,探索后训练阶段最优配置。
- 0%重叠时语义准确率比纯SFT提升10.4个百分点,100%重叠则无提升。
- 双指标评估揭示编译与语义差距超30%,传统方法难以发现此问题。
监督微调(SFT)后接组相对策略优化(GRPO)是常见的后训练流程。我们系统性地考察了SFT-GRPO数据重叠的影响,在六种仅训练方案不同的条件下,对未开启思考能力的Qwen3-8B模型进行Lean 4自动形式化任务的后训练评估:基础模型、仅SFT、仅GRPO,以及三种SFT+GRPO配置(GRPO提示与SFT语料重叠率为0%、30%、100%)。结果表明,保持SFT与GRPO数据完全分离能持续优于全重叠,且无需额外计算成本。在Gaokao-Formal和PutnamBench上,基于大语言模型判别的编译通过率和语义通过率均显示,重叠率越低,编译与语义准确率越高。在0%重叠下,GRPO相比SFT在Gaokao上带来10.4个百分点的语义提升;而100%重叠时,两项指标均无变化,表明GRPO阶段形同虚设。进一步分析发现,双指标评估揭示最高编译性能模型的语义差距超过30个百分点,这一差异在仅以编译通过率为标准时无法察觉。本研究首次将SFT-GRPO数据重叠作为后训练超参数进行受控探究,证明模型表现随训练阶段间数据共享程度显著变化。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) is a common post-training recipe. We conduct a controlled ablation over SFT-GRPO data overlap, evaluating Qwen3-8B (thinking disabled) post-trained for Lean 4 autoformalization under six conditions that differ solely in training recipe: a base model, SFT-only, GRPO-only, and three SFT+GRPO configurations where 0 percent, 30 percent, or 100 percent of the GRPO prompts coincide with the SFT corpus. Keeping SFT and GRPO data disjoint consistently outperforms full overlap at zero additional compute cost. Evaluating on Gaokao-Formal and PutnamBench under both compile pass at k and semantic pass at k assessed by an LLM judge, we find that lower overlap is monotonically associated with higher compilation and semantic accuracy. At 0 percent overlap, GRPO yields a 10.4 percentage point semantic gain over SFT alone on Gaokao, while at 100 percent overlap both metrics remain flat, rendering the GRPO stage effectively redundant. We further show that dual-metric evaluation reveals compile semantic gaps exceeding 30 percentage points for the highest compiling models, a disparity invisible under compile-only benchmarking. To our knowledge, this is the first controlled investigation of SFT-GRPO data overlap as a post-training hyperparameter, demonstrating how model behavior varies based on the degree of data sharing between training stages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。