从复杂强化学习模型中提取可解释策略，兼顾性能与透明度。
"So, Tell Me About Your Policy...": Distillation of interpretable policies from Deep Reinforcement Learning agents
- 基于优势函数构建可解释策略，保留专家行为特征
- 在经典控制和金融交易任务中实现高保真策略迁移
- 适合需要透明决策的医疗、金融等关键领域
深度强化学习（DRL）虽在Atari、围棋及金融交易等复杂任务中表现优异，但其黑箱特性导致难以理解状态特征重要性及策略生成机制。本文提出一种新算法，在理论保证下从复杂专家策略中蒸馏出可解释的线性策略，利用优势函数捕捉动作优劣原因。相比以往方法，本方案可在不重新收集数据的前提下，仅使用已有经验进行训练。在经典控制环境与金融交易场景中的实验表明，该方法能有效提取复杂策略中的关键信息，实现高性能与可解释性的平衡。
原文摘要 · Abstract (English)
Recent advances in Reinforcement Learning (RL) largely benefit from the inclusion of Deep Neural Networks, boosting the number of novel approaches proposed in the field of Deep Reinforcement Learning (DRL). These techniques demonstrate the ability to tackle complex games such as Atari, Go, and other real-world applications, including financial trading. Nevertheless, a significant challenge emerges from the lack of interpretability, particularly when attempting to comprehend the underlying patterns learned, the relative importance of the state features, and how they are integrated to generate the policy's output. For this reason, in mission-critical and real-world settings, it is often preferred to deploy a simpler and more interpretable algorithm, although at the cost of performance. In this paper, we propose a novel algorithm, supported by theoretical guarantees, that can extract an interpretable policy (e.g., a linear policy) without disregarding the peculiarities of expert behavior. This result is obtained by considering the advantage function, which includes information about why an action is superior to the others. In contrast to previous works, our approach enables the training of an interpretable policy using previously collected experience. The proposed algorithm is empirically evaluated on classic control environments and on a financial trading scenario, demonstrating its ability to extract meaningful information from complex expert policies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。