用预期收益与惊喜度之积来决定探索时机，更高效地利用探索预算。
Delightful Exploration
- 以惊喜度定义探索成本，仅在潜在回报高时才触发探索动作。
- 在伯努利、线性贝叶斯和表格型马尔可夫决策中，相同超参数无需调优即可适用。
- 相比Thompson采样和ε-贪心，探索未解问题时后悔值增长显著更慢。
大多数探索算法在不确定性被消除前广泛搜索。当动作空间过大无法在预算内解决时，从业者通常采用ε-贪心策略，虽限制了干扰但盲目覆盖。本文提出‘愉悦门控探索’（Delight-gated exploration, DE），一种主控-覆盖规则：仅当潜在愉悦度（预期改进乘以惊讶度）超过门限价格时，才执行探索动作。该实用启发式方法恢复了经典结果：庞达拉的代价搜索保留值规则，其中惊讶度决定了有效检查成本。已解决的臂退出门限，新臂在先验设定阈值以上关闭，选定的线性贝叶斯覆盖动作消耗有限信息预算。在伯努利贝叶斯、线性贝叶斯及表格型马尔可夫决策过程中，相同超参数无需重新调整，且在未解区域中，DE表现出远弱于Thompson Sampling和ε-贪心的后悔增长。愉悦度提升行动力的原因正是其提升学习效率：通过收益与惊讶度的乘积对稀缺资源定价。
原文摘要 · Abstract (English)
Most exploration algorithms search broadly until uncertainty is resolved. When the action space is too large to resolve within budget, practitioners default to $\varepsilon$-greedy, which bounds disruption but spends its override blindly. We introduce \textit{Delight-gated exploration} (DE), a host--override rule that spends exploratory actions only when their prospective delight (expected improvement times surprisal) exceeds a gate price. This practical heuristic recovers a classical result: Pandora's reservation-value rule for costly search, with surprisal setting the effective inspection cost. Resolved arms exit the gate, fresh arms shut off above a prior-determined threshold, and selected linear-bandit overrides consume finite information budget. Across Bernoulli bandits, linear bandits, and tabular MDPs, the same hyperparameters transfer without retuning, and DE shows much weaker regret growth than Thompson Sampling and $\varepsilon$-greedy in the tested unresolved regimes. Delight improves acting for the same reason it improves learning: it prices scarce resources by the product of upside and surprisal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。