用语言模型+树搜索自动设计奖励函数，提升复杂控制任务的优化效率。
RF-Agent: Automated Reward Function Design via Language Agent Tree Search
- 将语言模型视为智能体，用树搜索迭代优化奖励函数设计。
- 在17个低级控制任务中表现优于传统方法，显著提升性能。
- 适合对强化学习奖励设计效率有要求的研究者和工程师。
为低级控制任务设计高效奖励函数是一项挑战。近期研究尝试减少对专家经验的依赖，利用大语言模型（LLMs）结合任务信息生成密集奖励函数。这些方法通常以训练结果为反馈，通过贪心或进化算法迭代生成新奖励函数。然而，它们对历史反馈利用不足且搜索效率低下，在复杂控制任务中改进有限。为此，我们提出RF-Agent，将LLMs视为语言智能体，将奖励函数设计建模为序列决策过程，通过更好的上下文推理增强优化。该框架整合蒙特卡洛树搜索（MCTS），利用LLMs的多阶段上下文推理能力管理奖励设计与优化流程，更充分地利用历史信息并提升搜索效率，从而发现更有前景的奖励函数。在17个多样化的低级控制任务中，实验结果表明该方法具有显著有效性。源代码已公开于https://github.com/deng-ai-lab/RF-Agent。
原文摘要 · Abstract (English)
Designing efficient reward functions for low-level control tasks is a challenging problem. Recent research aims to reduce reliance on expert experience by using Large Language Models (LLMs) with task information to generate dense reward functions. These methods typically rely on training results as feedback, iteratively generating new reward functions with greedy or evolutionary algorithms. However, they suffer from poor utilization of historical feedback and inefficient search, resulting in limited improvements in complex control tasks. To address this challenge, we propose RF-Agent, a framework that treats LLMs as language agents and frames reward function design as a sequential decision-making process, enhancing optimization through better contextual reasoning. RF-Agent integrates Monte Carlo Tree Search (MCTS) to manage the reward design and optimization process, leveraging the multi-stage contextual reasoning ability of LLMs. This approach better utilizes historical information and improves search efficiency to identify promising reward functions. Outstanding experimental results in 17 diverse low-level control tasks demonstrate the effectiveness of our method. The source code is available at https://github.com/deng-ai-lab/RF-Agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。