用搜索优化奖励函数，让大模型数学推理能力提升近20%。
Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning
- 把奖励函数设计当成可优化目标，通过迭代生成与筛选提升性能。
- 五轮搜索后最佳集成模型在GSM8K上达F1 0.795，比基线高0.19。
- 反馈循环是关键，随机组合奖励会严重退化，证明优化有效。
数学推理是大语言模型的重要评估基准。强化学习是提升大模型推理能力的标准后训练方法，但性能仍高度依赖奖励函数的设计。本文提出一种搜索驱动的框架，将奖励函数本身作为优化对象。固定基础模型，仅优化奖励设定。候选奖励函数由前沿语言模型生成，经自动验证、在Llama-3.2-3B-Instruct基础上使用低秩适配（LoRA）进行500步组相对策略优化（GRPO）训练，并以GSM8K测试集上的F1分数排序。前一轮的排序摘要反馈至下一轮生成。经过五轮迭代共产生50个候选奖励。平均F1从第1轮的0.596提升至第5轮的0.632，最优个体达F1=0.787。评估七个顶级奖励的集成配置，最佳集成取得F1=0.795（95%置信区间[0.756, 0.832]），准确率0.660（[0.635, 0.686]），相比仅用基础奖励的GRPO基线（F1=0.609）提升0.19。成对McNemar检验结合邦弗朗尼校正显示，所有含五种及以上奖励的配置在α=0.05/21下无显著差异。三种子重新训练最佳集成得F1=0.785。随机抽取五种奖励的对照组骤降至F1=0.047，表明性能提升源于排名反馈机制，而非奖励数量叠加。
原文摘要 · Abstract (English)
Mathematical reasoning is a key benchmark for large language models. Reinforcement learning is a standard post-training mechanism for improving the reasoning capabilities of large language models, yet performance remains sensitive to the design of the reward function that drives policy optimization. This paper introduces a search-driven framework that treats the reward specification itself as an object of optimization. The setting of interest is one in which the base model is held fixed and the reward specification is the primary remaining design lever. Candidate reward functions are generated by a frontier language model, validated automatically, screened through 500-step Group Relative Policy Optimization (GRPO) training runs on a Llama-3.2-3B-Instruct base model with Low-Rank Adaptation (LoRA), and ranked by F1 on the GSM8K test set. Ranked summaries from prior rounds are then fed back into the next round of generation. Over five rounds, the search produces 50 candidate rewards. The mean F1 rises from 0.596 in Round 1 to 0.632 in Round 5, and the top individual reward reaches F1 = 0.787. Seven ensemble configurations of top-ranked rewards are evaluated. The best ensemble achieves F1 = 0.795 (95% bootstrap CI [0.756, 0.832]) and accuracy 0.660 [0.635, 0.686], a 0.19 absolute F1 gain over a base-rewards-only GRPO baseline (F1 = 0.609). Pairwise McNemar tests with Bonferroni correction show all five-or-more-reward configurations are statistically indistinguishable at α = 0.05/21. A three-seed re-training of the best ensemble yields F1 of 0.785. A randomly drawn 5-reward control collapses to F1 = 0.047, which shows that the ranked-feedback loop, not the additive signal of having more rewards, drives the gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。