提出战略关联度量化长期决策中的动作依赖,揭示哪些行动需协同执行才有效。
Strategically Linked Decisions in Long-Term Planning and Reinforcement Learning
- 用战略关联度衡量动作间依赖:若后续动作不可用,当前动作的采取概率下降多少
- 在黑箱强化学习中识别出关键联动动作对,解释其决策逻辑
- 适用于评估推荐系统是否可独立采纳,或需整体策略配合
长期规划(如强化学习)涉及制定策略:一系列共同服务于目标的动作,而非单独优化即时收益。某些动作虽牺牲短期利益,但为后续更高回报动作创造条件。这些动作仅在后续动作可用时才具价值。本文提出战略关联度:在约束后续动作不可用时,当前动作采取概率的下降程度,以此量化动作间的依赖关系。通过三个应用验证其有效性:(i) 解释黑箱强化学习代理的决策,识别其内部的战略联动对;(ii) 提升决策支持系统的最坏情况性能,区分推荐动作是可独立采纳的改进,还是需承诺更广策略才有效的战略联动;(iii) 仅通过干预测量战略关联度,刻画非强化学习代理的规划过程——以真实交通模拟器为例,通过道路封闭分析众多驾驶员涌现的路径行为的有效规划范围。
原文摘要 · Abstract (English)
Long-term planning, as in reinforcement learning (RL), involves finding strategies: actions that collectively work toward a goal rather than individually optimizing their immediate outcomes. As part of a strategy, some actions are taken at the expense of short-term benefit to enable future actions with even greater returns. These actions are only advantageous if followed up by the actions they facilitate, consequently, they would not have been taken if those follow-ups were not available. In this paper, we quantify such dependencies between planned actions with strategic link scores: the drop in the likelihood of one decision under the constraint that a follow-up decision is no longer available. We demonstrate the utility of strategic link scores through three practical applications: (i) explaining black-box RL agents by identifying strategically linked pairs among decisions they make, (ii) improving the worst-case performance of decision support systems by distinguishing whether recommended actions can be adopted as standalone improvements or whether they are strategically linked hence requiring a commitment to a broader strategy to be effective, and (iii) characterizing the planning processes of non-RL agents purely through interventions aimed at measuring strategic link scores - as an example, we consider a realistic traffic simulator and analyze through road closures the effective planning horizon of the emergent routing behavior of many drivers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。