让强化学习的复杂行为可解释，用数学定义行为模式并分析其成因。
BXRL: Behavior-Explainable Reinforcement Learning
- 将行为定义为跨多轮的行动模式，作为可度量的独立对象
- 提出对比行为框架，通过微分方法解析策略偏好原因
- 提供工具链支持行为测量与参数敏感性分析，适配现有解释方法
强化学习中，智能体常习得违背奖励设计的不良行为。可解释强化学习（XRL）能回答特定动作、轨迹或策略的解释问题，但缺乏对跨多轮实验的行为模式的正式定义。本文提出行为可解释强化学习（BXRL），将行为视为第一类对象，定义行为度量为任意函数 $m : Π o bR$，使用户可精确表达感兴趣的动作模式，并量化策略对它的表现强度。提出对比行为概念，将“为何选择动作 $a$ 而非 $a'$”简化为“为何 $m(π)$ 值高”，并通过微分探索该问题。本文未实现新解释方法，而是分析三种现有方法，并提出如何适配以解释行为。同时，将HighwayEnv驾驶环境移植至JAX，提供定义、度量和对模型参数求导行为的接口。
原文摘要 · Abstract (English)
A major challenge of Reinforcement Learning is that agents often learn undesired behaviors that seem to defy the reward structure they were given. Explainable Reinforcement Learning (XRL) methods can answer queries such as "explain this specific action", "explain this specific trajectory", and "explain the entire policy". However, XRL lacks a formal definition for behavior as a pattern of actions across many episodes. We provide such a definition, and use it to enable a new query: "Explain this behavior". We present Behavior-Explainable Reinforcement Learning (BXRL), a new problem formulation that treats behaviors as first-class objects. BXRL defines a behavior measure as any function $m : Π\to \mathbb{R}$, allowing users to precisely express the pattern of actions that they find interesting and measure how strongly the policy exhibits it. We define contrastive behaviors that reduce the question "why does the agent prefer $a$ to $a'$?" to "why is $m(π)$ high?" which can be explored with differentiation. We do not implement an explainability method; we instead analyze three existing methods and propose how they could be adapted to explain behavior. We present a port of the HighwayEnv driving environment to JAX, which provides an interface for defining, measuring, and differentiating behaviors with respect to the model parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。