提出双视角算法提升非平稳环境下的强化学习表现
Empirical Comparison of Forgetting Mechanisms for UCB-based Algorithms on a Data-Driven Simulation Platform
- 融合折扣与滑动窗口机制，兼顾长期与短期收益
- 滑动窗口法在动态环境中表现稳定，指数衰减法导致线性后悔
- 乐观集成策略是性能突破的关键，适合实时推荐场景
许多现实世界的带状问题涉及非平稳奖励分布，最优决策可能随环境变化而转移。然而，典型的多臂老虎机（MAB）模型如上置信界（UCB）算法在奖励分布随时间变化的非平稳环境中性能显著下降。为解决这一问题，本文提出并评估了FDSW-UCB——一种结合基于折扣的长期视角与基于滑动窗口的短期视角的新型双视图算法。构建了一个基于MovieLens-1M和Open Bandit数据集的数据驱动半合成仿真平台，用于测试算法在突变和渐变漂移场景下的适应能力。实验结果表明，配置良好的滑动窗口机制（SW-UCB）具有鲁棒性，而广泛使用的折扣方法（D-UCB）存在根本性学习失败，导致线性后悔。关键的是，采用乐观聚合策略的FDSW-UCB在动态设置中表现出色，凸显集成策略本身是成功的关键因素。
原文摘要 · Abstract (English)
Many real-world bandit problems involve non-stationary reward distributions, where the optimal decision may shift due to evolving environments. However, the performance of some typical Multi-Armed Bandit (MAB) models such as Upper Confidence Bound (UCB) algorithms degrades significantly in non-stationary environments where reward distributions change over time. To address this limitation, this paper introduces and evaluates FDSW-UCB, a novel dual-view algorithm that integrates a discount-based long-term perspective with a sliding-window-based short-term view. A data-driven semi-synthetic simulation platform, built upon the MovieLens-1M and Open Bandit datasets, is developed to test algorithm adaptability under abrupt and gradual drift scenarios. Experimental results demonstrate that a well-configured sliding-window mechanism (SW-UCB) is robust, while the widely used discounting method (D-UCB) suffers from a fundamental learning failure, leading to linear regret. Crucially, the proposed FDSW-UCB, when employing an optimistic aggregation strategy, achieves superior performance in dynamic settings, highlighting that the ensemble strategy itself is a decisive factor for success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。