用大模型辅助分解复杂任务，自动发现子任务并生成有效规则。
LLM-Augmented Symbolic Reinforcement Learning with Landmark-Based Task Decomposition
- 基于正负轨迹识别子任务，用一阶逻辑表示状态
- 大模型生成规则模板，经归纳逻辑编程优化为策略
- 减少对预设逻辑谓词的依赖，提升复杂任务求解能力
强化学习中的核心挑战之一是将复杂任务分解为更易学习的子任务。本文提出一种新方法，通过给定正负轨迹来识别子任务，并使用一阶谓词逻辑表示状态。我们设计了一种算法以自动检测子任务，随后利用大语言模型（LLM）生成实现每个子任务的逻辑规则模板。这些模板通过归纳逻辑编程（ILP）驱动的强化学习代理进一步微调为规则策略。实验表明，该算法能准确识别所有子任务。此外，评估显示大模型生成的常识性规则对解决子任务至关重要，显著降低了对环境中预定义一阶逻辑谓词的依赖。
原文摘要 · Abstract (English)
One of the fundamental challenges in reinforcement learning (RL) is to take a complex task and be able to decompose it to subtasks that are simpler for the RL agent to learn. In this paper, we report on our work that would identify subtasks by using some given positive and negative trajectories for solving the complex task. We assume that the states are represented by first-order predicate logic using which we devise a novel algorithm to identify the subtasks. Then we employ a Large Language Model (LLM) to generate first-order logic rule templates for achieving each subtask. Such rules were then further fined tuned to a rule-based policy via an Inductive Logic Programming (ILP)-based RL agent. Through experiments, we verify the accuracy of our algorithm in detecting subtasks which successfully detect all of the subtasks correctly. We also investigated the quality of the common-sense rules produced by the language model to achieve the subtasks. Our experiments show that our LLM-guided rule template generation can produce rules that are necessary for solving a subtask, which leads to solving complex tasks with fewer assumptions about predefined first-order logic predicates of the environment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。