让强化学习优化更灵活，能直接针对最差、最好或中间表现调整目标。
OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimation
- 通过排序统计量设计无偏梯度估计，可灵活优化不同分布目标
- 在有限样本下支持风险厌恶、抗异常值等目标，提升模型鲁棒性
- 只需替换奖励即可接入现有算法，适合高风险场景的强化学习
策略梯度方法通常优化期望回报，但许多实际应用关注回报的分布特性：尾部风险、异常值鲁棒性或最佳结果发现。本文提出 OrderGrad，一类基于似然比和重参数化的排序统计量梯度估计器。OrderGrad 能优化有限样本的 L-统计量，即排序后奖励或成本的加权平均，仅通过调整秩权重即可恢复诸如 VaR、CVaR、截尾均值、中位数以及前 m 项/最优 K 项等目标。对于任意固定样本量和秩权重向量，OrderGrad 提供对应排序统计量目标的无偏梯度估计。该方法实现为简单的奖励变换，可无缝集成到标准策略梯度或重参数化更新中。我们分析了估计器的方差行为，并在均值优化与部署目标不匹配的任务上进行了评估，包括大语言模型数学推理后训练等。OrderGrad 为风险规避、鲁棒及探索性学习提供统一、即插即用的解决方案。代码见：https://github.com/paavo5/ordergrad
原文摘要 · Abstract (English)
Policy-gradient methods usually optimize expected return, but many real world applications care about distributional properties of returns: tail risk, outlier robustness, or best-of-K discovery. We introduce OrderGrad, a family of likelihood-ratio and reparameterization gradient estimators for order-statistic objectives. OrderGrad optimizes finite-sample L-statistics, i.e., weighted averages of sorted rewards or costs, recovering objectives such as VaR, CVaR, trimmed means, medians, and top-m/best-of-K criteria by changing only the rank weights. For any fixed sample size and rank-weight vector, OrderGrad provides an unbiased gradient estimator for the corresponding order-statistic objective. The method is implemented as a simple reward transformation that can then be used in an otherwise standard policy-gradient or reparameterized update. We study the resulting estimator's variance behavior and evaluate it on tasks where mean optimization is mismatched to the deployment objective, including LLM math post-training and other tasks. OrderGrad provides a unified, plug-and-play route to risk-averse, robust, and exploratory learning. Code: https://github.com/paavo5/ordergrad
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。