用分阶段时间规划提升复杂时序逻辑任务的机器人控制成功率
TGPO: Temporal Grounded Policy Optimization for Signal Temporal Logic Tasks
- 将时序逻辑分解为带时间约束的子目标，分层优化策略
- 在五类环境中平均提升31.6%任务成功率，长时序任务表现更优
- 适合高维、长周期的机器人控制问题，如无人机与四足行走
学习复杂、长时程任务的控制策略是机器人与自主系统的核心挑战。信号时序逻辑（STL）提供了表达此类任务的强大语言，但其非马尔可夫特性及稀疏奖励使得标准强化学习难以求解。以往方法仅针对有限的STL片段，或使用STL鲁棒性作为稀疏终态奖励。本文提出时空对齐策略优化（TGPO），以解决通用STL任务。TGPO将STL分解为带时序的子目标与不变约束，构建分层框架：高层组件分配各子目标的具体时间，底层时序条件策略利用密集的阶段奖励信号完成序列子目标。推理时，通过采样多种时间分配并选择最优方案进行轨迹滚动。为高效学习多子目标复杂STL，引入已学判别器引导高层时间搜索，采用梅特罗波利斯-哈斯廷斯采样聚焦于时序可行解。我们在五个环境（从低维导航到操作、无人机、四足运动）中进行了实验。在广泛STL任务下，TGPO显著优于现有基线，尤其在高维与长时程场景中，平均任务成功率提升31.6%。代码将公开于 https://github.com/mengyuest/TGPO。
原文摘要 · Abstract (English)
Learning control policies for complex, long-horizon tasks is a central challenge in robotics and autonomous systems. Signal Temporal Logic (STL) offers a powerful and expressive language for specifying such tasks, but its non-Markovian nature and inherent sparse reward make it difficult to be solved via standard Reinforcement Learning (RL) algorithms. Prior RL approaches focus only on limited STL fragments or use STL robustness scores as sparse terminal rewards. In this paper, we propose TGPO, Temporal Grounded Policy Optimization, to solve general STL tasks. TGPO decomposes STL into timed subgoals and invariant constraints and provides a hierarchical framework to tackle the problem. The high-level component of TGPO proposes concrete time allocations for these subgoals, and the low-level time-conditioned policy learns to achieve the sequenced subgoals using a dense, stage-wise reward signal. During inference, we sample various time allocations and select the most promising assignment for the policy network to rollout the solution trajectory. To foster efficient policy learning for complex STL with multiple subgoals, we leverage the learned critic to guide the high-level temporal search via Metropolis-Hastings sampling, focusing exploration on temporally feasible solutions. We conduct experiments on five environments, ranging from low-dimensional navigation to manipulation, drone, and quadrupedal locomotion. Under a wide range of STL tasks, TGPO significantly outperforms state-of-the-art baselines (especially for high-dimensional and long-horizon cases), with an average of 31.6% improvement in task success rate compared to the best baseline. The code will be available at https://github.com/mengyuest/TGPO
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。