用大模型自动设计奖励函数，减少人工试错
Chain of Uncertain Rewards with Large Language Models for Reinforcement Learning
- 利用大模型分析文本语义，复用有效奖励组件
- 在9个IsaacGym环境和20项双臂操作任务中表现更优
- 降低奖励评估成本，适合需快速迭代的RL研究
设计有效的奖励函数是强化学习的核心挑战，传统方法依赖大量手动设计与评估，效率低且易忽略中间决策点的局部不确定性。为此，我们提出链式不确定奖励（CoUR）框架，结合大语言模型（LLM）实现奖励函数的自动化设计与评估。CoUR引入代码不确定性量化与相似性选择机制，通过文本与语义分析识别并复用最相关的奖励组件；通过解耦奖励项并采用贝叶斯优化，显著提升搜索效率与鲁棒性。我们在IsaacGym的9个原始环境及双臂操作基准的全部20个任务上进行了全面评估，结果表明CoUR不仅性能更优，还大幅降低了奖励评估成本。
原文摘要 · Abstract (English)
Designing effective reward functions is a cornerstone of reinforcement learning (RL), yet it remains a challenging and labor-intensive process due to the inefficiencies and inconsistencies inherent in traditional methods. Existing methods often rely on extensive manual design and evaluation steps, which are prone to redundancy and overlook local uncertainties at intermediate decision points. To address these challenges, we propose the Chain of Uncertain Rewards (CoUR), a novel framework that integrates large language models (LLMs) to streamline reward function design and evaluation in RL environments. Specifically, our CoUR introduces code uncertainty quantification with a similarity selection mechanism that combines textual and semantic analyses to identify and reuse the most relevant reward function components. By reducing redundant evaluations and leveraging Bayesian optimization on decoupled reward terms, CoUR enables a more efficient and robust search for optimal reward feedback. We comprehensively evaluate CoUR across nine original environments from IsaacGym and all 20 tasks from the Bidexterous Manipulation benchmark. The experimental results demonstrate that CoUR not only achieves better performance but also significantly lowers the cost of reward evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。