用正则化训练的随机性实现有效探索,无需额外设计探索策略。
RIE-Greedy: Regularization-Induced Exploration for Contextual Bandits
- 利用交叉验证正则化中的随机性,自然产生类似汤普森采样的探索行为。
- 在双臂老虎机场景下理论等价于汤普森采样,真实业务环境表现优于ε-贪婪等方法。
- 适合大规模工业场景,尤其适用于无法直接应用传统探索策略的黑箱模型。
现实世界中具有复杂奖励模型的上下文老虎机问题常通过迭代训练模型(如提升树)解决。然而,难以在这些黑箱估计器上直接应用简单有效的探索策略(如汤普森采样或UCB)。现有方法依赖复杂假设或不可行的计算过程,难以验证与实现。本文探索纯贪婪(无探索)动作选择策略,利用模型拟合过程本身的随机性作为内在探索来源。具体而言,我们发现基于交叉验证的正则化过程中的随机性可自然诱导类似汤普森采样的探索。理论证明该正则化诱导探索在双臂老虎机情况下等价于汤普森采样,并在大规模商业环境中实证表现出可靠探索能力,优于ε-贪婪及其他先进方法。本工作揭示了正则化估计器训练本身即可激发有效探索,为上下文老虎机设计提供理论洞见与实践指导。
原文摘要 · Abstract (English)
Real-world contextual bandit problems with complex reward models are often tackled with iteratively trained models, such as boosting trees. However, it is difficult to directly apply simple and effective exploration strategies--such as Thompson Sampling or UCB--on top of those black-box estimators. Existing approaches rely on sophisticated assumptions or intractable procedures that are hard to verify and implement in practice. In this work, we explore the use of an exploration-free (pure-greedy) action selection strategy, that exploits the randomness inherent in model fitting process as an intrinsic source of exploration. More specifically, we note that the stochasticity in cross-validation based regularization process can naturally induce Thompson Sampling-like exploration. We show that this regularization-induced exploration is theoretically equivalent to Thompson Sampling in the two-armed bandit case and empirically leads to reliable exploration in large-scale business environments compared to benchmark methods such as epsilon-greedy and other state-of-the-art approaches. Overall, our work reveals how regularized estimator training itself can induce effective exploration, offering both theoretical insight and practical guidance for contextual bandit design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。