让大模型自己生成奖励，动态提升多任务强化学习效果。
RLAR: An Agentic Reward System for Multi-task Reinforcement Learning on Large Language Models
- 用智能体自动从网络获取并合成最优奖励模型
- 在数学、编程等任务上提升10%到60%
- 适合需要自适应奖励的复杂多任务场景
基于强化学习的大语言模型对齐高度依赖奖励函数质量。然而，静态的、领域特定的奖励模型训练成本高，且在强化学习迭代过程中遇到分布外情况时泛化能力差。我们提出RLAR（从智能体奖励中学习），一种由智能体驱动的框架，可为每个查询动态分配定制化的奖励函数。具体而言，RLAR将奖励获取转化为动态工具合成与调用任务，利用大语言模型智能体自主从互联网检索最优奖励模型，并通过代码生成合成程序化验证器。这使得奖励系统能在训练过程中随数据分布变化而自我演化。实验表明，RLAR在数学、编程、翻译和对话任务上均实现10%至60%的性能提升。在RewardBench-V2上，显著优于静态基线，逼近性能上限，展现出通过动态奖励编排带来的优异泛化能力。数据与代码已开源：https://github.com/ZhuoerFeng/RLAR。
原文摘要 · Abstract (English)
Large language model alignment via reinforcement learning depends critically on reward function quality. However, static, domain-specific reward models are often costly to train and exhibit poor generalization in out-of-distribution scenarios encountered during RL iterations. We present RLAR (Reinforcement Learning from Agent Rewards), an agent-driven framework that dynamically assigns tailored reward functions to individual queries. Specifically, RLAR transforms reward acquisition into a dynamic tool synthesis and invocation task. It leverages LLM agents to autonomously retrieve optimal reward models from the Internet and synthesize programmatic verifiers through code generation. This allows the reward system to self-evolve with the shifting data distributions during training. Experimental results demonstrate that RLAR yields consistent performance gains ranging from 10 to 60 across mathematics, coding, translation, and dialogue tasks. On RewardBench-V2, RLAR significantly outperforms static baselines and approaches the performance upper bound, demonstrating superior generalization through dynamic reward orchestration. The data and code are available on this link: https://github.com/ZhuoerFeng/RLAR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。