用噪声对比学习让大模型更精准推理，效果媲美顶尖方法。
ReNCE: Learning to Reason by Noise Contrastive Estimation
- 将多个推理结果分为正负样本，直接优化正样本概率
- 在数学推理任务上达到与DAPO、在线DPO相当的性能
- 无需经验调参，自动区分好坏推理路径，适合想简化训练的人
GRPO是赋予预训练大模型推理能力的标准方法，通过估计一组K个结果的优势值，在信任域内提升优势为正的结果。由于其对优劣结果的区分较柔和，常需依赖如非对称裁剪和零方差数据过滤等额外改进，但这些技巧需大量经验且难发现。本文提出一种显式对比学习方法：不估计优势，而是将K个结果划分为正负集合，直接最大化正样本的似然。该方法可视为大模型推理中（多标签）噪声对比估计的在线实现。我们在一系列挑战性数学基准上验证了该方法，性能与DAPO及在线DPO等强基线相当。
原文摘要 · Abstract (English)
GRPO is a standard approach to endowing pretrained LLMs with reasoning capabilities. It estimates the advantage of an outcome from a group of $K$ outcomes, and promotes those with positive advantages inside a trust region. Since GRPO discriminates between good and bad outcomes softly, it benefits from additional refinements such as asymmetric clipping and zero-variance data filtering. While effective, these refinements require significant empirical insight and can be challenging to identify. We instead propose an explicit contrastive learning approach. Instead of estimating advantages, we bifurcate $K$ outcomes into positive and negative sets, then maximize the likelihood of positive outcomes. Our approach can be viewed as an online instantiation of (multi-label) noise contrastive estimation for LLM reasoning. We validate our method by demonstrating competitive performance on a suite of challenging math benchmarks against strong baselines such as DAPO and online DPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。