arXiv:2505.02069cs.LGstat.ML2025-05被引 4

用神经网络学奖励函数,突破维度瓶颈,更高效更稳定。

Neural Logistic Bandits

  • 设计新不等式,避开高维特征干扰
  • 理论证明后悔上界仅依赖有效维度
  • 适合高维神经网络强化学习场景

我们研究神经逻辑老虎机问题,目标是通过神经网络在逻辑链接函数下学习未知的奖励函数。现有方法或对κ(奖励分布最小方差的倒数)有不利依赖,或直接依赖特征维度d,而d在神经网络中可能极大。本文提出一种新型伯恩斯坦型不等式,适用于自归一化向量值鞅,可规避对环境维度的直接依赖。由此推导出后悔上界仅随有效维度˜d增长,且对κ依赖极小。基于此不等式,我们设计两种算法:NeuralLog-UCB-1和NeuralLog-UCB-2,分别保证˜O(˜d√(κT))和˜O(˜d√(T/κ))的后悔上界,优于现有结果。最后在合成与真实数据集上验证了理论发现。

原文摘要 · Abstract (English)

We study the problem of neural logistic bandits, where the main task is to learn an unknown reward function within a logistic link function using a neural network. Existing approaches either exhibit unfavorable dependencies on $κ$, where $1/κ$ represents the minimum variance of reward distributions, or suffer from direct dependence on the feature dimension $d$, which can be huge in neural network-based settings. In this work, we introduce a novel Bernstein-type inequality for self-normalized vector-valued martingales that is designed to bypass a direct dependence on the ambient dimension. This lets us deduce a regret upper bound that grows with the effective dimension $\widetilde{d}$, not the feature dimension, while keeping a minimal dependence on $κ$. Based on the concentration inequality, we propose two algorithms, NeuralLog-UCB-1 and NeuralLog-UCB-2, that guarantee regret upper bounds of order $\widetilde{O}(\widetilde{d}\sqrt{κT})$ and $\widetilde{O}(\widetilde{d}\sqrt{T/κ})$, respectively, improving on the existing results. Lastly, we report numerical results on both synthetic and real datasets to validate our theoretical findings.

强化学习神经网络带宽优化后悔分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。