在部分策略值可线性表示下,强化学习求最优策略是计算上困难的。
Computational Hardness of Reinforcement Learning with Partial $q^π$-Realizability
- 提出弱于策略值可表示但强于最优值可表示的新假设框架
- 证明在特定策略集下,学习ε最优策略是NP难且指数下界成立
- 揭示即使扩展策略集,计算难度仍无法避免,适合理论研究者
本文研究了一种新型线性函数逼近设定下的强化学习计算复杂性,称为部分 $q^π$-可表示性。该设定要求所有属于预定义策略集 $Π$ 的策略的价值函数均可线性表示,目标是在此假设下学习一个 $ε$-最优策略。该假设弱于 $q^π$-可表示性,但强于 $q^*$-可表示性,是一种自然出现函数逼近的实用模型。我们证明,在此设定下学习 $ε$-最优策略是计算上困难的:针对参数化贪婪策略集(argmax),建立了 NP-难性;在随机指数时间假设下,若策略集包含 softmax 策略,则特征向量维度上存在指数级下界(除非 NP = RP)。通过从 $δ$-Max-3SAT 与 $δ$-Max-3SAT(b) 问题归约到 GLinear-$κ$-RL(贪婪)与 SLinear-$κ$-RL(softmax)实例,证明了此类困难与 $q^*$-可表示性情形一致,表明即便 $Π$ 扩展至非最优策略,计算难度依然存在。结果表明,在部分 $q^π$-可表示性下,一般正向计算结果不可达,与 $q^π$-可表示性在生成访问模型下的结果形成对比。
原文摘要 · Abstract (English)
This paper investigates the computational complexity of reinforcement learning in a novel linear function approximation regime, termed partial $q^π$-realizability. In this framework, the objective is to learn an $ε$-optimal policy with respect to a predefined policy set $Π$, under the assumption that all value functions for policies in $Π$ are linearly realizable. The assumptions of this framework are weaker than those in $q^π$-realizability but stronger than those in $q^*$-realizability, providing a practical model where function approximation naturally arises. We prove that learning an $ε$-optimal policy in this setting is computationally hard. Specifically, we establish NP-hardness under a parameterized greedy policy set (argmax) and show that - unless NP = RP - an exponential lower bound (in feature vector dimension) holds when the policy set contains softmax policies, under the Randomized Exponential Time Hypothesis. Our hardness results mirror those in $q^*$-realizability and suggest computational difficulty persists even when $Π$ is expanded beyond the optimal policy. To establish this, we reduce from two complexity problems, $δ$-Max-3SAT and $δ$-Max-3SAT(b), to instances of GLinear-$κ$-RL (greedy policy) and SLinear-$κ$-RL (softmax policy). Our findings indicate that positive computational results are generally unattainable in partial $q^π$-realizability, in contrast to $q^π$-realizability under a generative access model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。