将深度强化学习策略转化为简洁可读的规则系统。
ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies
- 用斜向超平面划分状态空间，结合局部线性模型构建决策树。
- 提取的规则策略参数少且性能接近原模型，还能提升原模型表现。
- 适合需要解释性的工业场景或安全敏感应用。
可解释性仍是强化学习中的关键问题。当动作空间为连续时，从复杂环境中训练的智能体中提炼出可解释策略尤为困难。本文提出ORCAID，一种从混合连续-离散环境、连续动作空间下的深度强化学习智能体中提取可解释规则策略的新方法。核心贡献是一种高效的斜向决策树训练算法，通过超平面划分状态空间并拟合局部线性模型。其关键思想是三阶段分裂搜索：高效随机初始化、局部优化与反向消除。最后合并相邻叶节点，生成一组简洁的可解释规则以描述原始深度强化学习策略。我们在多个强化学习环境中评估了ORCAID，结果表明所提取的规则策略在参数量极低的情况下仍保持优异性能，甚至能进一步提升原始深度强化学习策略的表现。
原文摘要 · Abstract (English)
Explainability remains a key issue in reinforcement learning (RL). Distilling an interpretable policy from an agent trained in a complex environment is particularly challenging when the action space is continuous. We introduce ORCAID, a novel method for extracting interpretable rule-based policies from RL agents operating in mixed continuous-discrete environments with continuous action spaces. Our main contribution is an efficient oblique decision tree training algorithm that partitions the state space by hyperplanes and fits local linear models. The key idea lies in a three-stage split search: efficient random initialization, local refinement, and backward elimination. Finally, adjacent leaves are merged to yield a concise set of interpretable rules describing a given deep RL policy. We evaluate ORCAID across multiple RL environments, demonstrating that the extracted rule-based policies maintain strong performance with a low number of parameters and can even be used to improve the performance of the original deep RL policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。