让强化学习自动发现并终止目标,解决无明确反馈时的探测难题
Autonomous Goal Detection and Cessation in Reinforcement Learning: A Case Study on Source Term Estimation
- 引入自反馈机制,让智能体自主判断目标是否达成
- 在源项估计任务中,成功率、搜索时间等指标显著优于传统方法
- 适合复杂环境下的自主决策场景,如传感器网络探测
强化学习在动态环境中革新了决策能力,但在缺乏明确反馈信号时难以自主识别和完成目标。例如,在源项估计问题中,环境信息不完整导致难以提供清晰的反馈,也难以定义和评估源位置的确定过程。为此,我们提出自主目标检测与终止(AGDC)模块,通过引入自反馈机制,增强多种强化学习算法在目标未明时的自主目标检测与任务终止能力。该方法通过近似智能体的信念来有效识别并停止未知目标,显著提升了强化学习在低反馈环境中的表现。我们将AGDC集成至深度Q网络、近端策略优化和深度确定性策略梯度算法,并在源项估计任务上进行验证。实验结果表明,AGDC增强的算法在成功率、平均移动距离和搜索时间上均显著优于传统统计方法(如infotaxis、entrotaxis、双控制)及随机动作选择方法,展现出在复杂真实场景中的高效性与有效性。
原文摘要 · Abstract (English)
Reinforcement Learning has revolutionized decision-making processes in dynamic environments, yet it often struggles with autonomously detecting and achieving goals without clear feedback signals. For example, in a Source Term Estimation problem, the lack of precise environmental information makes it challenging to provide clear feedback signals and to define and evaluate how the source's location is determined. To address this challenge, the Autonomous Goal Detection and Cessation (AGDC) module was developed, enhancing various RL algorithms by incorporating a self-feedback mechanism for autonomous goal detection and cessation upon task completion. Our method effectively identifies and ceases undefined goals by approximating the agent's belief, significantly enhancing the capabilities of RL algorithms in environments with limited feedback. To validate effectiveness of our approach, we integrated AGDC with deep Q-Network, proximal policy optimization, and deep deterministic policy gradient algorithms, and evaluated its performance on the Source Term Estimation problem. The experimental results showed that AGDC-enhanced RL algorithms significantly outperformed traditional statistical methods such as infotaxis, entrotaxis, and dual control for exploitation and exploration, as well as a non-statistical random action selection method. These improvements were evident in terms of success rate, mean traveled distance, and search time, highlighting AGDC's effectiveness and efficiency in complex, real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。