提出新方法缓解离线到在线学习中的分布偏移问题,提升决策效率。
Sample-Mean Anchored Thompson Sampling for Offline-to-Online Learning with Distribution Shift

- 用中位数锚定机制融合在线、离线数据,修正分布偏移导致的估计偏差。
- 理论证明算法能安全利用离线数据,减少后悔值,且效果随离线数据量增大而提升。
- 适合有历史数据但环境变化的在线决策场景,如推荐系统、广告投放。
离线到在线学习旨在通过利用离线记录数据来改进在线决策。该任务的核心挑战是离线与在线环境之间的分布偏移。尽管已有研究尝试利用偏移的离线数据,但大多依赖于UCB类算法。汤普森采样(TS)是另一类经典强化学习算法,因其出色的实证表现和基于贝叶斯框架的天然适配性,在离线到在线学习中具有优势。然而,与UCB不同,TS中的后验样本并不保证对真实臂均值是乐观的,这使得仅基于在线或混合数据构建的索引难以比较和使用。为此,本文提出样本均值锚定的汤普森采样(Anchor-TS),引入一种基于中位数的锚定规则,将臂索引定义为在线后验样本、混合后验样本和在线样本均值的中位数。该中位数锚定机制系统性地纠正了由分布偏移引起的偏差,缓解了次优臂的过估计和最优臂的欠估计问题,同时在偏移较小时有效利用离线信息获得更精确的估计。我们建立了理论保证,证明所提算法可安全地利用离线数据加速在线学习,并量化了分布偏移程度与离线数据规模对后悔减少的影响。大量实验表明,该算法在多个基准上持续优于现有方法。
原文摘要 · Abstract (English)
Offline-to-online learning aims to improve online decision-making by leveraging offline logged data. A central challenge in this setting is the distribution shift between offline and online environments. While some existing works attempt to leverage shifted offline data, they largely rely on UCB-type algorithms. Thompson sampling (TS) represents another canonical class of bandit algorithms, well known for its strong empirical performance and naturally suited to offline-to-online learning through its Bayesian formulation. However, unlike UCB indices, posterior samples in TS are not guaranteed to be optimistic with respect to the true arm means. This makes indices constructed from purely online and hybrid data difficult to compare and complicates their use. To address this issue, we propose sample-mean anchored TS (Anchor-TS), which introduces a novel median-based anchoring rule that defines the arm index as the median of an online posterior sample, a hybrid posterior sample, and the online sample mean. The median anchoring systematically corrects bias induced by distribution shift by mitigating over-estimation for suboptimal arms and under-estimation for optimal arms, while exploiting offline information to obtain more accurate estimates when the shift is small. We establish theoretical guarantees showing that the proposed algorithm safely leverages offline data to accelerate online learning, and quantifying how the degree of distribution shift and the size of offline data affect the resulting regret reduction. Extensive experiments demonstrate consistent improvements of our algorithm over baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。