提出可复现的评估框架,对比变异数感知与经典强化学习算法表现。
A Framework for Fair Evaluation of Variance-Aware Bandit Algorithms
- 构建标准化实验环境,系统比较8种经典与变异数感知算法。
- 在高不确定性场景中,变异数感知算法显著优于传统方法。
- 适合关注算法公平比较与鲁棒性评估的研究者参考。
多臂赌博机(MAB)是复杂强化学习算法的基础,但其算法评估与对比仍面临缺乏标准化条件和可复现性的挑战,尤其对经典方法如UCB的变异数感知扩展而言,性能高度依赖于环境特性。本文提出一个可复现的评估框架,用于系统比较八种经典与变异数感知的MAB算法。该框架基于Bandit Playground代码库,包含明确的实验设置、多重性能指标(奖励、累计损失、奖励分布、风险价值、动作最优性),以及交互式分析界面,支持一致且透明的结果分析。结果显示,在高不确定性环境中,当各臂奖励差异微小时,变异数感知算法具有优势;而在可分性较强或经精细调参的场景下,经典算法表现相当甚至更优。本研究贡献包括:(1) 一套系统化的MAB算法评估框架;(2) 明确了变异数感知方法适用的具体条件。
原文摘要 · Abstract (English)
Multi-armed bandit (MAB) problems serve as a fundamental building block for more complex reinforcement learning algorithms. However, evaluating and comparing MAB algorithms remains challenging due to the lack of standardized conditions and replicability. This is particularly problematic for variance-aware extensions of classical methods like UCB, whose performance can heavily depend on the underlying environment. In this study, we address how performance differences between bandit algorithms can be reliably observed, and under what conditions variance-aware algorithms outperform classical ones. We present a reproducible evaluation designed to systematically compare eight classical and variance-aware MAB algorithms. The evaluation framework, implemented in our Bandit Playground codebase, features clearly defined experimental setups, multiple performance metrics (reward, regret, reward distribution, value-at-risk, and action optimality), and an interactive evaluation interface that supports consistent and transparent analysis. We show that variance-aware algorithms can offer advantages in settings with high uncertainty where the difficulty arises from subtle differences between arm rewards. In contrast, classical algorithms often perform equally well or better in more separable scenarios or if fine-tuned extensively. Our contributions are twofold: (1) a framework for systematic evaluation of MAB algorithms, and (2) insights into the conditions under which variance-aware approaches outperform their classical counterparts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。