用视觉语言模型自动设计更符合指令的强化学习奖励函数。
RDA: Reward Design Agent for Reinforcement Learning
- 基于视觉语言模型分解任务并分析轨迹，动态优化奖励代码
- 在12个桌面上操作任务中，指令对齐度显著优于基线方法
- 适合需要高语义对齐的机器人控制场景，如复杂指令执行
强化学习已实现令人瞩目的机器人技能，但通常依赖人工设计的奖励函数，耗时且难以与人类意图对齐。近期工作如Eureka通过大语言模型从任务描述迭代生成和优化奖励代码，但仅依赖成功率等粗粒度反馈，缺乏对行为语义的洞察，导致策略虽达成目标却常与指令不符。我们提出奖励设计智能体（RDA），一种基于视觉语言模型的代理框架，将语义理解注入奖励设计过程。RDA分解任务、可视化评估轨迹、总结失败模式，并迭代修正奖励代码以更好匹配任务指令。在来自ManiSkill的12个桌面上操作任务和HumanoidBench的4个全身操作任务上，RDA生成的策略显著更符合指令，同时保持与基线相当的任务成功率。视频及生成的奖励代码详见https://nitinkamra1992.github.io/reward-design-agent。
原文摘要 · Abstract (English)
Reinforcement learning has enabled the acquisition of impressive robotic skills, but typically requires hand-crafted reward functions that are slow to design and difficult to align with human intentions. Recent work, such as Eureka, automates reward design by using an LLM to iteratively generate and refine reward code from task descriptions. However, they rely on coarse feedback signals such as success rate, which provide little semantic insight into the learned behavior. As a result, their trained policies achieve the final goal but are frequently poorly aligned with task instructions. We introduce the Reward Design Agent (RDA), a VLM-based agentic framework that injects semantic understanding into reward design. RDA decomposes tasks, visually evaluates trajectories, summarizes failure modes, and iteratively revises reward code to better align with task instructions. Across 12 tabletop manipulation tasks from ManiSkill and 4 whole-body manipulation tasks from HumanoidBench, RDA produces policies substantially more instruction-aligned than those of other baselines, while achieving comparable task success rates. Videos and the generated reward code are available on https://nitinkamra1992.github.io/reward-design-agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。