用逻辑任务引导探索，让强化学习更快学会复杂控制。
Sample-Efficient Reinforcement Learning with Temporal Logic Objectives: Leveraging the Task Specification to Guide Exploration
- 根据任务逻辑公式设计探索方向，避开无效搜索。
- 在复杂任务中样本效率提升显著，比基线快数倍。
- 适合高不确定性、复杂目标的机器人控制场景。
本文研究在动态不确定且控制目标以线性时序逻辑（LTL）公式表达的系统中，如何高效学习最优控制策略。不确定性体现在工作空间结构和控制决策结果上，导致未知的马尔可夫决策过程（MDP）。现有针对LTL任务的强化学习算法通常采用均匀探索产品MDP状态空间（如ε-贪婪策略），牺牲了样本效率，尤其在奖励稀疏、MDP规模大或任务复杂时问题更严重。本文提出一种加速强化学习算法，通过新颖的任务驱动探索策略，引导探索向可能满足任务的方向进行，显著提升样本效率。我们提供了理论分析和大量对比实验，验证了方法的有效性；随着任务复杂度或MDP规模增大，优势愈发明显。
原文摘要 · Abstract (English)
This paper addresses the problem of learning optimal control policies for systems with uncertain dynamics and high-level control objectives specified as Linear Temporal Logic (LTL) formulas. Uncertainty is considered in the workspace structure and the outcomes of control decisions giving rise to an unknown Markov Decision Process (MDP). Existing reinforcement learning (RL) algorithms for LTL tasks typically rely on exploring a product MDP state-space uniformly (using e.g., an $ε$-greedy policy) compromising sample-efficiency. This issue becomes more pronounced as the rewards get sparser and the MDP size or the task complexity increase. In this paper, we propose an accelerated RL algorithm that can learn control policies significantly faster than competitive approaches. Its sample-efficiency relies on a novel task-driven exploration strategy that biases exploration towards directions that may contribute to task satisfaction. We provide theoretical analysis and extensive comparative experiments demonstrating the sample-efficiency of the proposed method. The benefit of our method becomes more evident as the task complexity or the MDP size increases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。