arXiv:2510.08794cs.LGcs.AI2025-10被引 1

伪装探索:在不被察觉前提下快速找到隐藏最优选项。

Deceptive Exploration in Multi-armed Bandits

  • 用KL散度约束伪装行为,控制被发现风险
  • 伪装探索速率上限为Θ(√T),受公开奖励差距限制
  • 算法自适应探索强度,适合隐蔽性要求高的场景

我们研究多臂赌博机场景,每把臂有公开和私有奖励分布。观察者预期代理按公开奖励使用Thompson采样,但伪装代理希望在不被察觉的情况下快速识别最优私有臂。观察者可观察公开奖励与选择的臂,但无法观测私有奖励;代理则同时知晓两者。我们将可探测性定义为实际抽臂概率与观察者预期概率之间的逐步Kullback-Leibler(KL)散度约束。将成功抽取公开次优臂建模为成功率随每次成功而下降的伯努利过程,证明在该KL约束下此类抽取最多以Θ(√T)速率发生。随后基于公开与私有均值构建极大极小问题,其解刻画了最优错误指数。最后提出一种受顶尖两算法启发的算法,能根据公开次优性差距自动调整探索强度。数值实验验证了Θ(√T)速率及算法行为。

原文摘要 · Abstract (English)

We consider a multi-armed bandit setting in which each arm has a public and a private reward distribution. An observer expects an agent to follow Thompson Sampling according to the public rewards, however, the deceptive agent aims to quickly identify the best private arm without being noticed. The observer can observe the public rewards and the pulled arms, but not the private rewards. The agent, on the other hand, observes both the public and private rewards. We formalize detectability as a stepwise Kullback-Leibler (KL) divergence constraint between the actual pull probabilities used by the agent and the anticipated pull probabilities by the observer. We model successful pulling of public suboptimal arms as a % Bernoulli process where the success probability decreases with each successful pull, and show these pulls can happen at most at a $Θ(\sqrt{T}) $ rate under the KL constraint. We then formulate a maximin problem based on public and private means, whose solution characterizes the optimal error exponent for best private arm identification. We finally propose an algorithm inspired by top-two algorithms. This algorithm naturally adapts its exploration according to the hardness of pulling arms based on the public suboptimality gaps. We provide numerical examples illustrating the $Θ(\sqrt{T}) $ rate and the behavior of the proposed algorithm.

多臂赌博机伪装探索隐蔽学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。