让智能体自动学会任务规则,解释性更强且性能更优。
Fully Learnable Neural Reward Machines
- 端到端学习符号映射与任务逻辑机,无需预设规则
- 在复杂任务中表现优于基于RNN的旧方法
- 兼具深度强化学习的易用性与可解释性
非马尔可夫强化学习任务要求智能体基于完整状态-动作轨迹做出决策。传统方法常借助线性时序逻辑(LTL)或自动机等符号形式化表达长期目标,但依赖预定义符号接地(SG)函数和先验任务知识。本文提出全可学习神经奖励机器(FLNRM),可端到端学习符号映射与自动机结构,无需任何先验知识。该方法应用方式如同经典深度强化学习(DRL),却因自动机的有限紧凑性而具备更强可解释性。实验表明,将FLNRM与DRL结合后,其性能超越基于循环神经网络(RNN)的现有方法。
原文摘要 · Abstract (English)
Non-Markovian Reinforcement Learning (RL) tasks present significant challenges, as agents must reason over entire trajectories of state-action pairs to make optimal decisions. A common strategy to address this is through symbolic formalisms, such as Linear Temporal Logic (LTL) or automata, which provide a structured way to express temporally extended objectives. However, these approaches often rely on restrictive assumptions -- such as the availability of a predefined Symbol Grounding (SG) function mapping raw observations to high-level symbolic representations, or prior knowledge of the temporal task. In this work, we propose a fully learnable version of Neural Reward Machines (NRM), which can learn both the SG function and the automaton end-to-end, removing any reliance on prior knowledge. Our approach is therefore as easily applicable as classic deep RL (DRL) approaches, while being far more explainable, because of the finite and compact nature of automata. Furthermore, we show that by integrating Fully Learnable Reward Machines (FLNRM) with DRL, our method outperforms previous approaches based on Recurrent Neural Networks (RNNs).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。