三种语言模型训练方法本质是调节同一参数，决定学习强度。
GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity
- 三方法均通过标准差调节单个数值，反映答案分歧程度。
- 分歧越大（半对半）更新越强，一致则无更新，符合群体标准差恒等式。
- 揭示了看似简单的归一化实为决定学习位置与力度的核心开关。
训练语言模型进行推理时，三种主流方法看似各异，实则均调整单一数值——标准差，反映提示词生成答案的分歧程度。模型对每道题多次作答，自动检查器标记对错，答案的分歧度由这些标记的标准差衡量：当对错各半时最大，全同则为零。群组相对策略优化（GRPO）对其取倒数，修正版GRPO（Dr. GRPO）取消该操作，解耦剪裁与动态采样策略优化（DAPO）则舍弃标准差为零的组。本文证明三者实为同一调节旋钮的三种设置。该旋钮非表面修饰：在对错奖励下，分歧度恰好等于训练更新量，即‘群体标准差恒等式’。分歧组学习最多，一致组无更新、静默。该结果同时确定高权重问题及所需尝试次数。实验在真实难题数据集Big-Math和受控训练中验证直觉。看似无害的归一化步骤，实为决定学习发生位置与强度的核心控制机制。
原文摘要 · Abstract (English)
Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt's sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。