通过增强负样本挖掘长思维链模型中的有效推理步骤
Unearthing Gems from Stones: Policy Optimization with Negative Sample Augmentation for LLM Reasoning
- 分步评估负样本，识别其中的反思与纠错环节
- 在相同数据集上优于基线，提升样本利用效率
- 适合需要高效训练长推理链的模型开发者
近期推理语言模型正从短思维链转向长思维链模式。由于长思维链模型的采样计算成本高，最大化固定训练数据集的利用率至关重要。我们分析发现，负样本中包含自省和错误修正等有价值成分，但现有方法要么完全丢弃负样本（RFT），要么对所有词元施加同等惩罚（RL），未能充分利用这些潜在学习信号。为此，我们提出行为约束策略梯度与负样本增强框架（BCPG-NSA），包含三个阶段：1）样本分割；2）结合大模型与评判模型进行共识性步骤正确性评估；3）设计负样本增强策略，以有效挖掘负样本中的正向推理步骤。实验表明，使用相同训练数据集，BCPG-NSA在多个挑战性数学与编程推理基准上均优于基线，提升样本效率，并在多轮迭代中展现鲁棒性与可扩展性。
原文摘要 · Abstract (English)
Recent advances in reasoning language models have witnessed a paradigm shift from short to long CoT pattern. Given the substantial computational cost of rollouts in long CoT models, maximizing the utility of fixed training datasets becomes crucial. Our analysis reveals that negative responses contain valuable components such as self-reflection and error-correction steps, yet primary existing methods either completely discard negative samples (RFT) or apply equal penalization across all tokens (RL), failing to leverage these potential learning signals. In light of this, we propose Behavior Constrained Policy Gradient with Negative Sample Augmentation (BCPG-NSA), a fine-grained offline RL framework that encompasses three stages: 1) sample segmentation, 2) consensus-based step correctness assessment combining LLM and PRM judgers, and 3) policy optimization with NSA designed to effectively mine positive steps within negative samples. Experimental results show that BCPG-NSA outperforms baselines on several challenging math/coding reasoning benchmarks using the same training dataset, achieving improved sample efficiency and demonstrating robustness and scalability when extended to multiple iterations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。