arXiv:2410.07533cs.LGstat.ML2024-10NeurIPS被引 7

研究噪声干扰下线性老虎机的鲁棒学习,揭示强弱干扰差异并解决错置率依赖问题。

Corruption-Robust Linear Bandits: Minimax Optimality and Gap-Dependent Misspecification

  • 区分强/弱干扰,建立统一分析框架
  • 首次给出对抗性线性老虎机的上下界匹配结果
  • 将抗干扰算法扩展至错置率依赖场景,实现理论突破

在线性老虎机中,当奖励受到干扰时,学习者如何有效学习?尽管已有大量研究,但对不同对抗模型和干扰度量的系统理解仍不充分,且最小最大后悔界限尚未完全刻画。本文比较了两种常见干扰类型:强干扰(干扰程度依赖于学习者选择的动作)与弱干扰(干扰程度独立于动作选择)。我们提出统一框架分析二者。针对随机线性老虎机,完全刻画了强弱干扰下的最小最大后悔差距。同时,首次研究受干扰的对抗性线性老虎机,获得上界与下界匹配的结果,其依赖关系与干扰水平一致。进一步揭示抗干扰学习与错置率依赖错置之间的联系——错置水平与动作次优性成正比,该设定由Liu等(2023a)首次提出。我们提出通用归约方法,使任意抗干扰算法可处理此类错置,并以黑箱方式恢复Liu等结果,显著推广至线性马尔可夫决策过程,首次获得强化学习中的错置率依赖结果。然而,此归约未达最优率。为此,我们设计专用算法,在线性老虎机中实现错置率依赖错置的最优边界,解答了Liu等(2023a)提出的开放问题。

原文摘要 · Abstract (English)

In linear bandits, how can a learner effectively learn when facing corrupted rewards? While significant work has explored this question, a holistic understanding across different adversarial models and corruption measures is lacking, as is a full characterization of the minimax regret bounds. In this work, we compare two types of corruptions commonly considered: strong corruption, where the corruption level depends on the action chosen by the learner, and weak corruption, where the corruption level does not depend on the action chosen by the learner. We provide a unified framework to analyze these corruptions. For stochastic linear bandits, we fully characterize the gap between the minimax regret under strong and weak corruptions. We also initiate the study of corrupted adversarial linear bandits, obtaining upper and lower bounds with matching dependencies on the corruption level. Next, we reveal a connection between corruption-robust learning and learning with gap-dependent mis-specification, a setting first studied by Liu et al. (2023a), where the misspecification level of an action or policy is proportional to its suboptimality. We present a general reduction that enables any corruption-robust algorithm to handle gap-dependent misspecification. This allows us to recover the results of Liu et al. (2023a) in a black-box manner and significantly generalize them to settings like linear MDPs, yielding the first results for gap-dependent misspecification in reinforcement learning. However, this general reduction does not attain the optimal rate for gap-dependent misspecification. Motivated by this, we develop a specialized algorithm that achieves optimal bounds for gap-dependent misspecification in linear bandits, thus answering an open question posed by Liu et al. (2023a).

线性老虎机鲁棒学习错置分析强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。