改进的探索策略让上下文强化学习更高效,尤其适合高维问题。
Feel-Good Thompson Sampling for Contextual Bandits: a Markov Chain Monte Carlo Showdown
- 引入乐观奖励机制,增强对高回报模型的探索
- 在线性与逻辑回归场景中表现优于传统方法,神经网络场景稍弱
- 代码开源,适合做现代上下文泛化基准测试
Thompson Sampling(TS)广泛用于解决上下文强化学习中的探索与利用权衡问题,但最新理论表明其在高维场景下探索不足。本文提出的Feel-Good TS(FG-TS)通过添加乐观奖励项,偏向高回报模型,在后验分布精确时可实现渐近最小最大最优遗憾。然而其在近似后验(常见于大规模或神经网络场景)下的表现尚未系统评估。我们首次在十一组真实与合成基准上系统比较了FG-TS及其平滑变体(SFG-TS)。通过对比精确后验(线性与逻辑回归带宽)与快速但粗略的随机梯度采样生成的近似后验,发现:当样本准确时大奖励项有益,但采样噪声主导时反而有害。在多数线性与逻辑回归任务中FG-TS优于基础TS,但在神经网络任务中表现较弱。尽管如此,因其性能稳定且易于使用,仍建议作为现代上下文带宽基准的默认基线。所有实验代码已公开于https://github.com/SarahLiaw/ctx-bandits-mcmc-showdown。
原文摘要 · Abstract (English)
Thompson Sampling (TS) is widely used to address the exploration/exploitation tradeoff in contextual bandits, yet recent theory shows that it does not explore aggressively enough in high-dimensional problems. Feel-Good Thompson Sampling (FG-TS) addresses this by adding an optimism bonus that biases toward high-reward models, and it achieves the asymptotically minimax-optimal regret in the linear setting when posteriors are exact. However, its performance with \emph{approximate} posteriors -- common in large-scale or neural problems -- has not been benchmarked. We provide the first systematic study of FG-TS and its smoothed variant (SFG-TS) across eleven real-world and synthetic benchmarks. To evaluate their robustness, we compare performance across settings with exact posteriors (linear and logistic bandits) to approximate regimes produced by fast but coarse stochastic-gradient samplers. Ablations over preconditioning, bonus scale, and prior strength reveal a trade-off: larger bonuses help when posterior samples are accurate, but hurt when sampling noise dominates. FG-TS generally outperforms vanilla TS in linear and logistic bandits, but tends to be weaker in neural bandits. Nevertheless, because FG-TS and its variants are competitive and easy-to-use, we recommend them as baselines in modern contextual-bandit benchmarks. Finally, we provide source code for all our experiments in https://github.com/SarahLiaw/ctx-bandits-mcmc-showdown.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。