用自适应分批策略利用多步前瞻信息，提升强化学习效率
Reinforcement Learning with Multi-Step Lookahead Information Via Adaptive Batching
- 根据当前状态动态划分未来信息批次，优化决策
- 理论证明算法的后悔上界接近最优，仅与前瞻长度ℓ相关
- 适合在已知未来几步信息的场景中使用，如机器人控制
我们研究具有多步前瞻信息的表格型强化学习问题。在采取动作前，学习者可观察ℓ步未来的转移和奖励实现：即在任意可能行动路径下，智能体将到达的状态及获得的奖励。虽然此类信息能显著提升价值，但寻找最优策略是NP难的。通常采用两种可处理的启发式方法：固定批次大小的分批策略和模型预测控制。我们首先指出这两种方法的局限性，并提出利用前瞻信息进行自适应（状态依赖）批次划分；这类策略称为自适应分批策略（ABPs）。我们推导了这些策略的最优贝尔曼方程，并设计了一种乐观最小后悔算法，可在未知环境中学习最优的ABP。我们的后悔上界在阶上是最优的，最多相差前瞻时长ℓ的一个因子，而ℓ通常可视为一个小常数。
原文摘要 · Abstract (English)
We study tabular reinforcement learning problems with multiple steps of lookahead information. Before acting, the learner observes $\ell$ steps of future transition and reward realizations: the exact state the agent would reach and the rewards it would collect under any possible course of action. While it has been shown that such information can drastically boost the value, finding the optimal policy is NP-hard, and it is common to apply one of two tractable heuristics: processing the lookahead in chunks of predefined sizes ('fixed batching policies'), and model predictive control. We first illustrate the problems with these two approaches and propose utilizing the lookahead in adaptive (state-dependent) batches; we refer to such policies as adaptive batching policies (ABPs). We derive the optimal Bellman equations for these strategies and design an optimistic regret-minimizing algorithm that enables learning the optimal ABP when interacting with unknown environments. Our regret bounds are order-optimal up to a potential factor of the lookahead horizon $\ell$, which can usually be considered a small constant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。