揭示选项机制在强化学习中失效的三大原因，指出盲目加选项反而有害。
When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Critic
- 终止规则学习无效，常导致立即退出，等同于不使用选项。
- 策略陷入僵化（称作‘政策坏死’），超五成状态无法更新动作选择。
- 多选项提升性能实因降低集体失败概率，从59%降至4%。
Option-critic 通过学习子策略及切换回主策略的时机规则来构建选项。本文从理论与实验两方面解释其性能随选项增加而提升的现象。首先，所学终止规则实际无用：当策略与终止条件读取相同值时，规则每步都触发，等价于始终终止；若策略探索而终止条件不探索（如 Option-critic 本身），规则可能阻碍探索，导致 Ω(T) 的遗憾，而始终终止仅产生 O(log T) 遗憾。强制每步终止可保持选项数量曲线不变。其次，选项内部策略几乎不探索，导致状态锁定首次选中的动作且不再更新，称为‘政策坏死’。我们提出状态级检测方法，发现典型选项中有五分之三的状态处于坏死状态。恢复探索可修复这些状态，一个选项即可完成任务。第三，新增选项并非提升单个选项性能，而是显著降低所有选项在同一状态同时失败的概率，由59%降至4%，性能随之提升。
原文摘要 · Abstract (English)
Option-critic learns options: sub-policies together with a learned rule for when each one hands control back. Its headline result is that performance improves as options are added. We explain that result, with theory and experiment. First, the termination rule option-critic learns by maximising return contributes nothing. When the termination test and the policy that picks options read the same values, the test fires at every step, so the learned rule is identical to always terminating. When that policy explores and the test does not, as in option-critic itself, the rule can block the exploration; there are instances where it suffers $\Omega(T)$ regret while always terminating holds to $O(\log T)$. Forcing termination at every step leaves the option-count curve intact. Second, the policy inside an option barely explores at all, so a state locks onto the first action that looked good and never updates again. We name this policy necrosis, give a state-level test for it, and find three fifths of states necrotic in a typical option. Restoring exploration repairs those states, and one option then solves the task. Third, extra options improve no option; what falls is the chance that all of them fail in the same state, from $59\\%$ to $4\\%$, and performance follows that joint quantity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。