用历史数据和任务描述优化大模型设计奖励，提升强化学习效果
Boosting Universal LLM Reward Design through Heuristic Reward Observation Space Evolution
- 通过表格缓存历史状态使用与成功率，突破大模型对话的马尔可夫限制
- 在基准任务上显著提升奖励设计效果与稳定性，验证框架有效性
- 适合做自动化强化学习奖励设计的研究者与工程师参考
大语言模型（LLMs）凭借其常识推理和代码生成能力，正成为自动化强化学习（RL）奖励设计的有力工具。通过与RL代理对话，LLMs构建奖励观察空间（ROS），选择相关环境状态并定义内部操作。然而，现有框架未能有效利用历史探索数据或人工任务描述来迭代演化该空间。本文提出一种新颖启发式框架，通过基于表格的探索缓存机制和文本-代码对齐策略，提升LLM驱动的奖励设计。框架引入状态执行表，追踪环境状态的历史使用频率与成功概率，克服了传统LLM对话中的马尔可夫约束，促进更高效的探索。此外，通过结构化提示将用户提供的任务描述与专家定义的成功标准对齐，确保奖励目标一致性。在多个基准强化学习任务上的综合评估表明，该框架具有显著的效果与稳定性。代码与视频演示见 jingjjjjjie.github.io/LLM2Reward。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are emerging as promising tools for automated reinforcement learning (RL) reward design, owing to their robust capabilities in commonsense reasoning and code generation. By engaging in dialogues with RL agents, LLMs construct a Reward Observation Space (ROS) by selecting relevant environment states and defining their internal operations. However, existing frameworks have not effectively leveraged historical exploration data or manual task descriptions to iteratively evolve this space. In this paper, we propose a novel heuristic framework that enhances LLM-driven reward design by evolving the ROS through a table-based exploration caching mechanism and a text-code reconciliation strategy. Our framework introduces a state execution table, which tracks the historical usage and success rates of environment states, overcoming the Markovian constraint typically found in LLM dialogues and facilitating more effective exploration. Furthermore, we reconcile user-provided task descriptions with expert-defined success criteria using structured prompts, ensuring alignment in reward design objectives. Comprehensive evaluations on benchmark RL tasks demonstrate the effectiveness and stability of the proposed framework. Code and video demos are available at jingjjjjjie.github.io/LLM2Reward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。