arXiv:2606.11968cs.LGstat.ML2026-06

提出高效算法EOFD-MLogB,解决高维多分类强化学习的计算瓶颈。

Efficient Multinomial Logistic Bandit via Frequent Directions

  • 用频数方向矩阵压缩降低参数估计和奖励更新复杂度
  • 每轮计算时间从立方级降至二次级,空间需求大幅减少
  • 适合高维动作空间的在线决策场景,如推荐系统

本文研究多项式逻辑老虎机(MLogB)的高效在线算法,其反馈分布为基于d维动作向量的K+1类多项式逻辑模型。现有典型算法OFUL-MLogB虽有$ ilde{ ext{O}}(Kd oot{T})$的遗憾界,但每轮需$ ext{O}(K^3d^3)$时间与$ ext{O}(K^2d^2)$空间,难以应对高维场景。为此,我们提出EOFD-MLogB,将频数方向矩阵压缩融入OFUL-MLogB。通过维护累积海森矩阵的低秩奇异值分解(SVD)压缩,将参数估计中的约束牛顿更新与奖励置信区间中的$Kd imes K$谱范数计算,分别简化为一维求根与$K imes K$特征值问题。由此实现每轮时间复杂度$ ext{O}(Kd(m+K)^2)$、空间复杂度$ ext{O}(Kd(m+K))$,其中$m \\< d$为压缩大小。理论证明其遗憾界为$ ilde{ ext{O}}(Δ_T(Kd\lnΔ_T+m) oot{T})$,当海森矩阵近似低秩时,$Δ_T$可控,遗憾接近原算法。实验验证了其计算效率与性能竞争力。

原文摘要 · Abstract (English)

This paper studies efficient online algorithms for multinomial logistic bandits (MLogB), where the feedback distribution over $K+1$ outcomes follows a multinomial logistic model of $d$-dimensional action vectors. A representative UCB-type algorithm, OFUL-MLogB, achieves a regret bound of $\tilde{\mathcal{O}}(Kd\sqrt{T})$, but still requires $\mathcal{O}(K^3d^3)$ time and $\mathcal{O}(K^2d^2)$ space per round due to parameter estimation and optimistic reward construction, which is prohibitive in high-dimensional settings. To address this limitation, we propose EOFD-MLogB, which integrates frequent directions matrix sketching into OFUL-MLogB. By maintaining a low-rank SVD sketch of the accumulated Hessian, constrained online Newton updates in parameter estimation and $Kd \times K$ spectral-norm computations in the reward bonus are reduced to one-dimensional root-finding tasks and $K \times K$ eigenvalue computations, respectively. This yields dominant per-round time complexity $\mathcal{O}(Kd(m+K)^2)$ and space complexity $\mathcal{O}(Kd(m+K))$, where $m \ll d$ is the sketch size. We further prove a regret bound of $\tilde{\mathcal{O}}(Δ_T(Kd\lnΔ_T+m)\sqrt{T})$, where the sketching error factor $Δ_T$ is controlled by the $m$-truncated spectral tail of the Hessian. Thus, when the Hessian is approximately low-rank, the regret is close to that of OFUL-MLogB. Experiments validate the computational efficiency and competitive performance.

强化学习在线优化矩阵压缩高维决策

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。