用自动机结构让强化学习零样本满足复杂时序任务
DeepLTL: Learning to Efficiently Satisfy Complex LTL Specifications for Multi-Task RL
- 基于布奇自动机建模LTL语义，动态生成满足策略
- 零样本通过率超现有方法，支持无限时域任务
- 适合需高可靠时序控制的机器人、自动驾驶场景
线性时序逻辑（LTL）近年被用于多任务强化学习中表达复杂时序任务。然而，学习在训练中未见的任意规范下高效满足目标仍具挑战。现有方法常仅适用于有限时域片段，局限于次优解，且未能充分处理安全约束。本文提出新学习方法，利用布奇自动机显式表示LTL语义，学习基于真值序列的条件策略以满足目标公式。在多种离散与连续环境中实验表明，该方法能零样本满足广泛有限与无限时域规范，在满足概率与效率上均优于现有方法。代码已公开：https://deep-ltl.github.io/
原文摘要 · Abstract (English)
Linear temporal logic (LTL) has recently been adopted as a powerful formalism for specifying complex, temporally extended tasks in multi-task reinforcement learning (RL). However, learning policies that efficiently satisfy arbitrary specifications not observed during training remains a challenging problem. Existing approaches suffer from several shortcomings: they are often only applicable to finite-horizon fragments of LTL, are restricted to suboptimal solutions, and do not adequately handle safety constraints. In this work, we propose a novel learning approach to address these concerns. Our method leverages the structure of Büchi automata, which explicitly represent the semantics of LTL specifications, to learn policies conditioned on sequences of truth assignments that lead to satisfying the desired formulae. Experiments in a variety of discrete and continuous domains demonstrate that our approach is able to zero-shot satisfy a wide range of finite- and infinite-horizon specifications, and outperforms existing methods in terms of both satisfaction probability and efficiency. Code available at: https://deep-ltl.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。