arXiv:2510.26000cs.LG2025-10NeurIPS

提出一种低频探索框架,让线性贝叶斯优化更安全高效

Infrequent Exploration in Linear Bandits

  • 设定探索周期,多数时间用贪婪策略,仅在固定时刻执行探索
  • 探索频率超对数阈值时,理论证明可达到最优后悔率
  • 适配任意探索算法,适合高成本或高风险场景

我们研究线性贝叶斯中的低频探索问题,填补了完全自适应探索(如UCB和Thompson Sampling)与纯贪婪策略之间的空白。前者可能每步都需探索,在安全关键或高成本领域不切实际;后者则需强多样性假设才能成功。为此,我们提出一个简单实用的框架INFEX:按预定周期执行基础探索策略,其余时间采用贪婪决策。尽管结构简单,理论分析表明,只要探索频率超过对数级阈值,INFEX即可实现与最优算法相当的实例相关后悔率。该框架通用且模块化,可无缝集成任意自适应探索方法,提升计算效率。实验验证其在后悔率和运行时间上均优于现有方法。

原文摘要 · Abstract (English)

We study the problem of infrequent exploration in linear bandits, addressing a significant yet overlooked gap between fully adaptive exploratory methods (e.g., UCB and Thompson Sampling), which explore potentially at every time step, and purely greedy approaches, which require stringent diversity assumptions to succeed. Continuous exploration can be impractical or unethical in safety-critical or costly domains, while purely greedy strategies typically fail without adequate contextual diversity. To bridge these extremes, we introduce a simple and practical framework, INFEX, explicitly designed for infrequent exploration. INFEX executes a base exploratory policy according to a given schedule while predominantly choosing greedy actions in between. Despite its simplicity, our theoretical analysis demonstrates that INFEX achieves instance-dependent regret matching standard provably efficient algorithms, provided the exploration frequency exceeds a logarithmic threshold. Additionally, INFEX is a general, modular framework that allows seamless integration of any fully adaptive exploration method, enabling wide applicability and ease of adoption. By restricting intensive exploratory computations to infrequent intervals, our approach can also enhance computational efficiency. Empirical evaluations confirm our theoretical findings, showing state-of-the-art regret performance and runtime improvements over existing methods.

强化学习贝叶斯优化低频探索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。