改进大模型推理时的奖励估计，提升自适应学习稳定性与效果。
Distribution-Aware Reward Estimation for Test-Time Reinforcement Learning
- 基于全路径分布而非单一多数结果进行奖励估计
- 在AIME 2024上实现25.3%的性能提升，AMC上提升5.3%
- 适合需要高可靠奖励信号的测试时强化学习场景
测试时强化学习（TTRL）使大语言模型能在无标注输入上自我优化，但其效果高度依赖于无真实标签监督下的奖励信号估计。现有方法多采用多次采样后多数投票（MV）生成确定性奖励，隐含假设多数路径提供可靠学习信号。我们发现该假设脆弱：MV将路径分布压缩为单一结果，丢弃非多数但正确路径的信息，导致系统性偏差。为此，提出分布感知奖励估计（DARE），将奖励估计从单一多数结果转向完整经验路径分布。DARE进一步引入探索奖励与分布剪枝机制，促进非多数路径探索并去噪奖励信号，实现更丰富、鲁棒的奖励估计。在多个挑战性推理基准上的实验表明，DARE显著提升优化稳定性和最终性能，在AIME 2024上相对基线提升25.3%，在AMC上提升5.3%。
原文摘要 · Abstract (English)
Test-time reinforcement learning (TTRL) enables large language models (LLMs) to self-improve on unlabeled inputs, but its effectiveness critically depends on how reward signals are estimated without ground-truth supervision. Most existing TTRL methods rely on majority voting (MV) over rollouts to produce deterministic rewards, implicitly assuming that the majority rollout provides a reliable learning signal. We show that this assumption is fragile: MV reduces the rollout distribution into a single outcome, discarding information about non-majority but correct actions candidates, and yields systematically biased reward estimates. To address this, we propose Distribution-AwareReward Estimation (DARE), which shifts reward estimation from a single majority outcome to the full empirical rollout distribution. DARE further augments this distribution-based reward with an exploration bonus and a distribution pruning mechanism for non-majority rollout exploration and reward denoise, yielding a more informative and robust reward estimation. Extensive experiments on challenging reasoning benchmarks show that DARE improves optimization stability and final performance over recent baselines, achieving relative improvements of 25.3% on challenging AIME 2024 and 5.3% on AMC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。