提出一种风险调整的老虎机算法,实现夏普比率优化的最优后悔上界。
Order Optimal Regret Bounds for Sharpe Ratio Optimization under Thompson Sampling
- 基于贝叶斯框架设计新算法,统一处理不同风险偏好下的探索。
- 理论证明后悔上界为O(log n),且与信息论下界匹配,达到最优。
- 适用于金融投资等需平衡收益与波动的决策场景。
本文研究在随机多臂老虎机(MAB)框架下最大化夏普比率(SR)的序贯决策问题。与传统累计奖励最大化的设定不同,SR优化需同时考虑期望收益与收益方差,目标函数为分式形式,依赖于收益分布的均值和方差。为此,我们提出夏普比率汤普森采样算法(SRTS),一种用于风险调整探索的贝叶斯方法。针对高斯奖励模型,该算法采用正态-伽马共轭后验,同时建模每条臂的均值与精度不确定性。与需区分风险策略的加性均值-方差(MV)方法不同,分式形式的SR目标可导出统一的采样规则,适用于所有风险容忍度。理论上,我们构建了适配SR目标的后悔分解,并提出解耦方法,分离均值与方差不确定性的贡献。该框架有效控制了后验中正态均值样本与伽马精度样本的交互作用。由此,我们建立了有限时间、分布依赖的$ \mathcal{O}(\log n)$期望后悔上界。进一步通过测度变换法推导出匹配的信息论下界,表明所提算法为阶最优。合成环境实验验证了 exttt{SRTS}的性能,结果表明其在多种风险-收益设置下均优于现有风险感知老虎机算法。
原文摘要 · Abstract (English)
In this paper, we study sequential decision-making for maximizing the Sharpe ratio (SR) in a stochastic multi-armed bandit (MAB) setting. Unlike standard bandit formulations that maximize cumulative reward, SR optimization requires balancing expected return and reward variability. As a result, the learning objective depends jointly on the mean and variance of the reward distribution and takes a fractional form. To address this problem, we propose the Sharpe Ratio Thompson Sampling \texttt{SRTS}, a Bayesian algorithm for risk-adjusted exploration. For Gaussian reward models, the algorithm employs a Normal-Gamma conjugate posterior to capture uncertainty in both the mean and the precision of each arm. In contrast to additive mean-variance (MV) formulations, which often require different algorithms across risk regimes, the fractional SR objective yields a single sampling rule that applies uniformly across risk tolerances. On the theoretical side, we develop a regret decomposition tailored to the SR objective and introduce a decoupling approach that separates the contributions of mean and variance uncertainty. This framework allows us to control the interaction between the Gaussian mean samples and the Gamma precision samples arising in the posterior. Using these results, we establish a finite-time distribution-dependent $\mathcal{O}(\log n)$ upper bound on the expected regret. We further derive a matching information-theoretic lower bound using a change-of-measure argument, showing that the proposed algorithm is order-optimal. Finally, experiments on synthetic bandit environments illustrate the performance of \texttt{SRTS} and demonstrate improvements over existing risk-aware bandit algorithms across a range of risk-return settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。