让多目标强化学习模型解释为何选某动作，只需微调偏好指令即可。
Command-Space Counterfactual Explanations for Pareto-Conditioned Reinforcement Learning
- 通过最小化指令变化，找到让智能体改选动作的条件。
- 提出CF-ZOO方法，在黑盒环境下高效定位关键指令调整。
- 解释结果用用户语言表达，适合需要可解释决策的场景。
Pareto Conditioned Networks（PCN）通过单一策略根据期望回报指令学习多种多目标强化学习行为，但其从指令与状态到动作的映射机制仍不透明。本文提出命令空间反事实解释：在固定状态和原指令下，搜索一个使同一训练好的策略选择指定替代动作的最小指令变更。贡献包括：第一，将解释形式化为仅依赖回报的指令干预，避免时域条件带来的歧义；第二，借鉴对抗机器学习方法用于强化学习解释；第三，提出边界种子方向搜索，相比纯局部优化更有效，形成新方法CF-ZOO。生成的解释具有可操作性，并以用户自身偏好语言表达，例如：‘若你的权衡略微偏向X，代理会选Y。’
原文摘要 · Abstract (English)
Pareto Conditioned Networks learn multiple multi-objective reinforcement learning behaviours by conditioning a single policy on a desired return command. However, the local mapping from command and state to action remains opaque. We propose command-space counterfactual explanations for PCNs: given a fixed state, original command, and foil action, we search, in a black-box setting, for a minimally changed desired-return command under which the same trained policy would choose the foil. Our contributions are threefold. First, we formulate PCN explanations as return-command interventions, using a return-only PCN variant that avoids the added ambiguity of horizon-conditioning. Second, we adapt adversarial machine learning methods to reinforcement-learning explanations. Third, we introduce a boundary-seeded directional search that improves over purely local optimization in the command-action landscape, resulting in our proposed approach CF-ZOO. The resulting explanations are actionable and intuitively expressed in the user's own preferences: "If your trade-off had shifted slightly towards X, the agent would have chosen Y."
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。