用符号公式直接解析环境输出，让强化学习更易用且可解释。
Reinforcement Learning with Symbolic Reward Machines
- 用符号公式直接处理环境观测，无需人工标注标签
- 在多个任务上超越传统强化学习方法，媲美已有奖励机性能
- 适合需要可解释性与低人工干预的RL应用
奖励机器（Reward Machines, RMs）是强化学习中表示稀疏、时序扩展任务的有效机制，尤其适用于非马尔可夫奖励场景。但其依赖环境提供的高阶标签信息，需用户为每项任务手动设计标签函数，限制了在主流强化学习框架中的应用。本文提出符号奖励机器（Symbolic Reward Machines, SRMs），并配套提出QSRM和LSRM学习算法。SRMs仅使用环境标准输出，通过符号公式作为守卫直接处理观测数据，无需人工标注标签。实验表明，所提方法在多个任务上优于基线强化学习模型，并达到现有奖励机器方法的性能水平。同时，该方法兼容通用环境定义，能为用户提供可解释的任务表达。
原文摘要 · Abstract (English)
Reward Machines (RMs) are an established mechanism in Reinforcement Learning (RL) to represent and learn sparse, temporally extended tasks with non-Markovian rewards. RMs rely on high-level information in the form of labels that are emitted by the environment alongside the observation. However, this concept requires manual user input for each environment and task. The user has to create a suitable labeling function that computes the labels. These limitations lead to poor applicability in widely adopted RL frameworks. We propose Symbolic Reward Machines (SRMs) together with the learning algorithms QSRM and LSRM to overcome the limitations of RMs. SRMs consume only the standard output of the environment and process the observation directly through guards that are represented by symbolic formulas. In our evaluation, our SRM methods outperform the baseline RL approaches and generate the same results as the existing RM methods. At the same time, our methods adhere to the widely used environment definition and provide interpretable representations of the task to the user.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。