arXiv:2504.03926cs.LGcs.SY2025-04被引 1

提出无需探索的线性带多臂老虎机方法,适用于超参优化场景。

An Exploration-free Method for a Linear Stochastic Bandit Driven by a Linear Gaussian Dynamical System

  • 基于卡尔曼滤波预测选择动作,不进行主动探索。
  • 理论分析表明性能依赖于系统可观测性,可实现低累积损失。
  • 适合高动作数、少迭代的超参优化问题,如强化学习训练。

在随机多臂老虎机中,学习者面临探索与利用的权衡。近年来,无探索方法(仅选择预测最优回报的动作)在线性带多臂问题中受到关注。本文提出一种新的线性带多臂设置:奖励是线性高斯动态系统的输出。该设定源于强化学习中的超参数优化问题——动作数远大于训练迭代次数。为此,我们提出卡尔曼滤波可观测性依赖探索(KODE)方法,利用卡尔曼滤波预测来选择动作。主要贡献在于对所提方法性能的理论分析,其表现依赖于底层线性高斯动态系统的可观测性。通过两种指标评估:累积遗憾(即最高可能回报与实际获得回报的期望差值之和),以及动作对齐度(衡量所选动作与系统状态变量的匹配程度)。为理解性能,我们证明了KODE隐式根据系统可观测性引导学习者选择动作。实验对比多个经典多臂老虎机算法,验证了理论结果。

原文摘要 · Abstract (English)

In stochastic multi-armed bandits, a major problem the learner faces is the trade-off between exploration and exploitation. Recently, exploration-free methods -- methods that commit to the action predicted to return the highest reward -- have been studied from the perspective of linear bandits. In this paper, we introduce a linear bandit setting where the reward is the output of a linear Gaussian dynamical system. Motivated by a problem encountered in hyperparameter optimization for reinforcement learning, where the number of actions is much higher than the number of training iterations, we propose Kalman filter Observability Dependent Exploration (KODE), an exploration-free method that utilizes the Kalman filter predictions to select actions. Our major contribution of this work is our analysis of the performance of the proposed method, which is dependent on the observability properties of the underlying linear Gaussian dynamical system. We evaluate KODE via two different metrics: regret, which is the cumulative expected difference between the highest possible reward and the reward sampled by KODE, and action alignment, which measures how closely KODE's chosen action aligns with the linear Gaussian dynamical system's state variable. To provide intuition on the performance, we prove that KODE implicitly encourages the learner to explore actions depending on the observability of the linear Gaussian dynamical system. This method is compared to several well-known stochastic multi-armed bandit algorithms to validate our theoretical results.

带多臂卡尔曼滤波超参优化无探索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。