计算资源越多，强化学习模型越能解决复杂任务并更好泛化。
On the Role of Computation in Reinforcement Learning
- 提出可变计算量的极简架构，分离参数与计算量
- 相同参数下，增加计算量显著提升性能和长序列任务泛化能力
- 适合追求高效计算利用的RL研究者与工程师
强化学习策略的计算资源如何影响其学习？固定参数规模的策略是否仍能从更多计算中获益？现有标准框架无法形式化回答这些问题。实践中，深度强化学习策略常以静态神经网络结构实现，导致参数量与计算量混淆。本文形式化定义了计算受限策略，并证明计算量更大的策略能解决更复杂问题，且在长时序任务上具备更强泛化能力。基于算法学习与无模型规划的前期工作，我们提出一种最小化架构，支持可变计算量使用。实验在31个不同任务（涵盖在线与离线强化学习）上验证理论：(1) 仅通过增加计算量，该架构即获得更强性能；(2) 在长序列测试任务上表现优于标准前馈网络或深度残差网络，即使后者参数量最多高出5倍。
原文摘要 · Abstract (English)
How does the amount of compute available to a reinforcement learning (RL) policy affect its learning? Can policies using a fixed amount of parameters, still benefit from additional compute? The standard RL framework does not provide a language to answer these questions formally. Empirically, deep RL policies are often parameterized as neural networks with static architectures, conflating the amount of compute and the number of parameters. In this paper, we formalize compute bounded policies and prove that policies which use more compute can solve problems and generalize to longer-horizon tasks that are outside the scope of policies with less compute. Building on prior work in algorithmic learning and model-free planning, we propose a minimal architecture that can use a variable amount of compute. Our experiments complement our theory. On a set 31 different tasks spanning online and offline RL, we show that $(1)$ this architecture achieves stronger performance simply by using more compute, and $(2)$ stronger generalization on longer-horizon test tasks compared to standard feedforward networks or deep residual network using up to 5 times more parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。