用模块化池子让强化学习奖励函数可复用、更稳定。
MLREF: Efficient Module Reuse for Reward Design in Reinforcement Learning via Large Language Models
- 构建可积累的奖励模块池，动态优化组件
- 17个任务上性能提升25.2%（行走）和6.6%（操作）
- 适合需要稳定迭代的RL奖励设计场景
奖励函数设计仍是强化学习中的瓶颈。尽管大语言模型（LLMs）实现了自动化奖励生成，但现有方法将奖励函数视为整体程序，难以可靠地保留和重用早期迭代中发现的有效组件，导致各轮性能不稳定。为此，我们提出模块级奖励演化框架（MLREF）。其核心是一个持久的模块池，作为可重用奖励组件的仓库。MLREF以模块池为主要优化目标：通过积累成功模块、优化表现不佳的模块并复用已验证组件，实现池的迭代进化；同时，奖励函数由从该池中选取的模块线性组合而成。为驱动这一演化过程，MLREF引入三种机制：基于反思的优化、混合信用分配和带回滚的合并策略，显著提升了奖励优化的有效性与鲁棒性。在17个任务上的实验表明，MLREF在行走任务上优于强基线25.2%，在操作任务上提升6.6%，且优化过程更稳定。
原文摘要 · Abstract (English)
Reward function design remains a bottleneck in reinforcement learning. While large language models (LLMs) have enabled automated reward generation, existing methods generate and revise reward functions as monolithic programs, making it difficult to reliably preserve and reuse effective components discovered in earlier iterations, leading to unstable performance across iterations. To address this, we propose Module Level Reward Evolution Framework (MLREF). At the core of MLREF is a module pool, a persistent repository of reusable reward components. MLREF treats the module pool as the primary optimization object: the pool evolves across iterations by accumulating successful modules, refining underperforming ones, and reusing proven components; while reward functions are constructed as linear combinations of modules drawn from this pool. To drive this evolution, MLREF integrates three mechanisms: reflection-based refinement, hybrid credit assignment, and a merge strategy with rollback, which together improve the effectiveness and robustness of reward optimization. Experiments on 17 tasks show that MLREF outperforms strong baselines by 25.2% in locomotion and 6.6% in manipulation, with more stable optimization dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。