通过分解未来预期结果,让强化学习决策过程更透明可解释。
Explainable Reinforcement Learning via Temporal Policy Decomposition
- 将策略分解为每个时间步的未来预期结果,揭示动作影响的时间序列。
- 在不同状态-动作对上生成对比性解释,准确展现策略的未来轨迹。
- 适合关注RL决策逻辑、需优化奖励函数的研究者与工程师。
我们从时间维度探究强化学习策略的可解释性,聚焦单个动作带来的未来结果序列。在强化学习中,价值函数压缩了跨多轨迹及无限时域的奖励信息,实现知识的紧凑表达,但这种压缩掩盖了序列决策中的时间细节,成为可解释性的关键挑战。本文提出时间策略分解(TPD),一种新颖的可解释性方法,通过期望未来结果(EFO)解释每个强化学习动作。该方法将广义价值函数分解为预测时域内每个时间步的EFO序列,揭示特定结果预期出现的时间点。我们采用固定时域时序差分学习,设计了一种离策略方法,用于学习最优与次优动作的EFO,支持不同状态-动作对之间的对比解释。实验表明,TPD生成的解释准确清晰,能够阐明策略的未来策略与预期轨迹,并帮助理解奖励构成,从而辅助调整奖励函数以符合人类期望。
原文摘要 · Abstract (English)
We investigate the explainability of Reinforcement Learning (RL) policies from a temporal perspective, focusing on the sequence of future outcomes associated with individual actions. In RL, value functions compress information about rewards collected across multiple trajectories and over an infinite horizon, allowing a compact form of knowledge representation. However, this compression obscures the temporal details inherent in sequential decision-making, presenting a key challenge for interpretability. We present Temporal Policy Decomposition (TPD), a novel explainability approach that explains individual RL actions in terms of their Expected Future Outcome (EFO). These explanations decompose generalized value functions into a sequence of EFOs, one for each time step up to a prediction horizon of interest, revealing insights into when specific outcomes are expected to occur. We leverage fixed-horizon temporal difference learning to devise an off-policy method for learning EFOs for both optimal and suboptimal actions, enabling contrastive explanations consisting of EFOs for different state-action pairs. Our experiments demonstrate that TPD generates accurate explanations that (i) clarify the policy's future strategy and anticipated trajectory for a given action and (ii) improve understanding of the reward composition, facilitating fine-tuning of the reward function to align with human expectations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。