用神经网络预测引导路径,让机器人快速学会最优行走策略。
Neural-Network-Driven Reward Prediction as a Heuristic: Advancing Q-Learning for Mobile Robot Path Planning
- 用双输出神经网络生成最优路径指引和次优区域,作为启发信息。
- 使Q-learning收敛速度提升90%,路径质量优于已有改进方法。
- 适合需要快速学习的移动机器人路径规划场景。
Q-learning是解决路径规划问题的常用强化学习方法,依赖智能体与环境交互以学习最大化累积奖励的最优策略。尽管广泛应用,其在实际中仍存在收敛缓慢的问题。为此,本文提出NDR-QL方法,利用神经网络输出作为启发信息加速Q-learning收敛。具体地,改进双输出神经网络模型,引入起点终点通道分离机制并增强特征融合。训练后,该模型可输出窄分布的最优概率(称作‘指引’)与宽分布的次优分布(称作‘区域’)。基于指引计算连续奖励函数,基于区域对Q表进行偏差初始化。在公开数据集上进行训练、验证与路径规划仿真实验。结果表明,NDR模型预测准确率较先前方法最高提升5%;所提NDR-QL方法使基准Q-learning收敛速度提升90%,并在路径质量指标上超越已有改进方法。
原文摘要 · Abstract (English)
Q-learning is a widely used reinforcement learning technique for solving path planning problems. It primarily involves the interaction between an agent and its environment, enabling the agent to learn an optimal strategy that maximizes cumulative rewards. Although many studies have reported the effectiveness of Q-learning, it still faces slow convergence issues in practical applications. To address this issue, we propose the NDR-QL method, which utilizes neural network outputs as heuristic information to accelerate the convergence process of Q-learning. Specifically, we improved the dual-output neural network model by introducing a start-end channel separation mechanism and enhancing the feature fusion process. After training, the proposed NDR model can output a narrowly focused optimal probability distribution, referred to as the guideline, and a broadly distributed suboptimal distribution, referred to as the region. Subsequently, based on the guideline prediction, we calculate the continuous reward function for the Q-learning method, and based on the region prediction, we initialize the Q-table with a bias. We conducted training, validation, and path planning simulation experiments on public datasets. The results indicate that the NDR model outperforms previous methods by up to 5\% in prediction accuracy. Furthermore, the proposed NDR-QL method improves the convergence speed of the baseline Q-learning method by 90\% and also surpasses the previously improved Q-learning methods in path quality metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。