无需环境模型，实时组合障碍物与目标策略完成安全导航。
Composable Model-Free RL for Navigation with Input-Affine Systems
- 基于输入仿射系统推导连续时间最优控制方程，获得动作二次型优势函数。
- 通过梯度下降学习静态/动态障碍物的避障策略，组合后保证碰撞规避。
- 适合复杂动态场景下需实时决策的自主机器人导航任务。
随着自主机器人进入复杂动态的真实环境，必须能够实时安全导航，但预测所有可能行为不可行。本文提出一种可组合的无模型强化学习方法：为每个环境元素（如目标或障碍物）分别学习价值函数和最优策略，并在线组合实现抵达目标与避障。假设未知的非线性连续时间动力学且为输入仿射型，我们推导出连续时间的哈密顿-雅可比-贝尔曼（HJB）方程，证明对应的优势函数在动作上呈二次型，最优策略亦可解析表达。基于此结构，设计了一种无模型的演员-评论家算法，利用梯度下降学习静态或移动障碍物的策略与价值函数。随后通过二次约束二次规划（QCQP）组合多个到达/避障模型，在价值函数等值面层面提供形式化避障保证，成为无需模型的CLF/CBF控制器替代方案。仿真显示其性能优于应用于离散时间近似的PPO基线。
原文摘要 · Abstract (English)
As autonomous robots move into complex, dynamic real-world environments, they must learn to navigate safely in real time, yet anticipating all possible behaviors is infeasible. We propose a composable, model-free reinforcement learning method that learns a value function and an optimal policy for each individual environment element (e.g., goal or obstacle) and composes them online to achieve goal reaching and collision avoidance. Assuming unknown nonlinear dynamics that evolve in continuous time and are input-affine, we derive a continuous-time Hamilton-Jacobi-Bellman (HJB) equation for the value function and show that the corresponding advantage function is quadratic in the action and optimal policy. Based on this structure, we introduce a model-free actor-critic algorithm that learns policies and value functions for static or moving obstacles using gradient descent. We then compose multiple reach/avoid models via a quadratically constrained quadratic program (QCQP), yielding formal obstacle-avoidance guarantees in terms of value-function level sets, providing a model-free alternative to CLF/CBF-based controllers. Simulations demonstrate improved performance over a PPO baseline applied to a discrete-time approximation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。