提出新方法CurveRL，让大模型推理更准。
CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning
- 基于通过率分布的排名与密度重加权提示
- 在多个基准上优于GRPO等基线方法
- 适合研究强化学习与大模型推理的学者
上下文或提示级别的重加权已成为增强大语言模型推理能力的关键技术，但最优加权的原则尚不明确。本文将提示重加权建模为在通过率函数空间中效用泛函的泛函导数，构建统一的最优性框架，兼容REINFORCE和GRPO等现有方法。在此基础上，提出一种分布感知的提示重加权方法CurveRL，基于分位数坐标变换，使每个提示的权重取决于其通过率的排名与密度，而非绝对值，从而反映学习动态中的通过率分布结构。在多个基准上的实验表明，CurveRL始终优于GRPO及其他RLVR基线方法。研究揭示了上下文分布控制是分析和设计提示重加权强化学习算法的理论轴心。代码已开源：https://github.com/zhyzmath/CurveRL。
原文摘要 · Abstract (English)
Context or prompt-level reweighting has emerged as a central algorithmic lever in Reinforcement Learning with Verified Rewards (RLVR) for improving the reasoning capability of large language models, yet the principle determining what constitutes an optimal weighting remains poorly understood. We address this gap by formulating prompt reweighting as a functional derivative of a utility functional defined in the pass-rate function space, yielding a unified optimality framework that accommodates existing schemes, including REINFORCE and GRPO. Building on this optimality framework, we propose a distribution-aware prompt reweighting approach, called CurveRL, based on a quantile coordinate transform, in which the weight assigned to each prompt depends not on the absolute value of pass rates but on its rank and density to reflect the distributional structure of the pass rates in the learning dynamics. Extensive experiments across multiple benchmarks demonstrate that our proposed CurveRL consistently outperforms GRPO and other RLVR baselines. Our study identifies context-distribution control as a principled axis for analyzing and designing prompt-reweighted RLVR algorithms. The code is released in https://github.com/zhyzmath/CurveRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。