用符号结构提升智能体完成组合任务的效率与鲁棒性
Reward Shaping and Action Masking for Compositional Tasks using Behavior Trees and LLMs
- 设计可反应失败、模块化配置的奖励与动作掩码结构MRBT
- 实验显示训练效率和任务成功率显著优于基线方法
- 适合需要灵活适应对象变化的复杂任务场景
将复杂任务分解为一系列简单子任务可提升智能体学习效率。强化学习（RL）可用于优化策略完成子任务，但依赖明确的子任务奖励并受益于动作掩码。现有工作利用大语言模型（LLMs）自动化实现奖励塑造和动作掩码，但未充分解决子任务失败时的反应能力及对不同物体的模块化支持问题。为此，我们提出掩码奖励行为树（MRBT），一种用于实时响应失败和适应不同对象的符号化奖励与动作掩码函数。设计了MRBT模板，并推导逻辑规范以构建与验证针对多物体交互子任务的MRBT。进一步开发自动化流程：使用LLM生成对任务对象变化鲁棒的MRBT，SMT求解器验证规范正确性，神经符号强化学习循环训练智能体完成组合任务。实验成功生成并优化了五个MRBT，一致提升训练效率与任务成功率，优于基线及无动作掩码的MRBT。此外，强调了三个优势：可迁移性、模块化与可验证性。
原文摘要 · Abstract (English)
Decomposing complex tasks into a sequence of simpler subtasks can improve learning efficiency for an autonomous agent. Reinforcement learning (RL) can be used to optimize agent policies to complete subtasks, but requires well-defined subtask rewards and benefits from action masking. Recent work uses large language models (LLMs) to automate reward shaping and action masking, however none of them fully address reactivity to subtask failure and modularity to varying objects for compositional tasks. To overcome these challenges, we develop masking reward behavior tree (MRBT), a symbolic structure used as a reactive and modular reward and action mask function. We design an MRBT template and derive logical specifications to construct and verify MRBTs for a sequence of object-interaction subtasks. Further, we develop an automated pipeline that uses an LLM to generate MRBTs robust to varying task objects, an SMT-solver to verify correctness of specifications, and a neurosymbolic RL loop to train agents on compositional tasks. Experiments demonstrate successful generation and refinement of five MRBTs, consistently improving training efficiency and task success rates over baselines and MRBTs without action masking. We further highlight three advantages of MRBTs: transferability, modularity, and verifiability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。