用批评者引导的分块量化，把复杂强化学习模型变成可解释的线性策略。
Critic-Driven Voronoi-Quantization for Distilling Deep RL Policies to Explainable Models
- 基于原始策略的评判网络，动态划分状态空间为多个区域。
- 在低价值区域添加新子策略，提升对原模型的逼近能力。
- 适合需要可解释性强且性能稳定的强化学习应用。
尽管已有多种深度强化学习策略解释方法，但在性能与可解释性之间仍难以平衡，且难选合适的代理模型。传统蒸馏仅最小化原模型与代理模型的行为距离，忽略了动作价值等强化学习特有成分。为此，本文提出一种无模型依赖的新方法——批评者驱动的Voronoi状态分区。该方法利用原始策略的评判网络，迭代地在价值不足的区域引入新的子策略，以衡量策略复杂度。分区采用Voronoi量化器，通过最近邻查找为状态空间中每一点分配一个线性函数，形成类似细胞的图示结构。我们在多个知名基准上验证该方法，证明其能用合理数量的线性函数逼近原策略。
原文摘要 · Abstract (English)
Despite many successful attempts at explaining Deep Reinforcement Learning policies using distillation, it remains difficult to balance the performance-interpretability trade-off and select a fitting surrogate model. In addition to this, traditional distillation only minimizes the distance between the behavior of the original and the surrogate policy while other RL-specific components such as action value are disregarded. To solve this, we introduce a new model-agnostic method called Critic-Driven Voronoi State Partitioning, which partitions a black box control policy into regions where a simple class of model can be optimized using gradient descent. By exploiting the critic value network of the original policy, we iteratively introduce new subpolicies in regions with insufficient value, standing in for a measure of policy complexity. The partitioning, a Voronoi quantizer, uses nearest neighbor lookups to assign a linear function to each point in the state space resulting in a cell-like diagram. We validate our approach on several well known benchmarks and proof that this distillation approaches the original policy using a reasonable sized set of linear functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。