用树形结构可视化强化学习决策路径，帮人理解智能体为何这么选。
SPOTting the Future: Lookahead Explanations for Deep Reinforcement Learning
- 通过采样动作构建可解释的有限时域决策树，不依赖具体模型。
- 能准确还原策略中最高概率动作，且在高熵策略下行为可分析。
- 适合交通信号控制等复杂决策场景，比单步特征分析更全面。
深度强化学习（DRL）智能体在复杂环境中表现优异，但其决策过程难以解释。我们提出SPOT（Sampling Policy Observation Tree），一种无需依赖模型、基于采样的解析框架。在拥有策略和环境模拟器的前提下，SPOT通过采样动作并递归模拟后续状态，构建一个可解释的有限时域树，以实证方式呈现策略的动作偏好及其可能的后续演化。我们提供了形式化保证，证明了SPOT在渐近条件下可恢复策略的唯一最可能动作，并刻画了高熵策略下的分歧行为。我们在SUMO-RL交通信号控制任务中验证了SPOT的有效性。案例研究表明，其树状表示可用于检查策略偏好、比较不同未来轨迹，并揭示单步特征归因方法无法发现的下游行为。
原文摘要 · Abstract (English)
Deep reinforcement learning (DRL) agents achieve strong performance in complex environments, yet their decision-making processes remain difficult to interpret. We introduce SPOT (Sampling Policy Observation Tree), a novel model-agnostic, sampling-based framework for interpreting DRL policies. Given access to the policy and an environment simulator, SPOT constructs an interpretable finite-horizon tree by sampling actions and recursively simulating the resulting successor states. The tree provides an empirical representation of the policy's action preferences and their possible downstream evolution. We provide formal guarantees establishing SPOT's asymptotic recovery of the policy's unique most probable action and characterizing its disagreement behavior under high-entropy policies. We demonstrate SPOT in the SUMO-RL traffic-signal control domain. The case study illustrates how its tree-based representation can be used to inspect policy preferences, compare alternative future trajectories, and reveal downstream behaviors that are not visible through single-timestep feature-attribution methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。