让智能体学会在实时决策中自适应思考时间，提升表现。
Finding the Time to Think: Learning Planning Budgets in Real-Time RL
- 用轻量门控策略动态决定每个状态下的思考时长。
- 在5个实时游戏上超越固定预算和启发式方法。
- 可跨不同硬件部署，适合实时决策场景应用。
思考需要时间。在实时环境中，时间并非免费。标准强化学习（RL）通过让环境无限等待智能体决策来回避此问题。本文研究真实时间强化学习环境，其中环境在等待智能体动作时持续推进。基于已有实时形式化，我们提出可变延迟实时强化学习，允许智能体在每个决策点自主选择思考时长。对于所用规划代理，最优思考时间是状态相关的，盲目规划思考时间可能导致智能体停滞。为此，我们在规划器之上训练一个轻量级门控策略，以根据状态选择合适的规划预算。在实时Pac-Man、Tetris、Snake、Speed Hex和Speed Go上，该门控策略优于固定预算和启发式基线，并可成功迁移至环境与智能体分别运行于两块GPU的实时设置中。
原文摘要 · Abstract (English)
Deliberating takes time. In real-time settings, that time is not free. Standard reinforcement learning (RL) sidesteps this as the environment waits indefinitely for the agent's decision. Instead, we study real-time RL environments where the environment progresses while waiting for the agent's action. Building on prior real-time formalizations, we introduce variable-delay real-time RL, where the agent chooses how long to deliberate at each decision point since the environment progresses. For the planning agents we use, the right delay is state-dependent, and naively planning how long to plan can paralyze the agent. We instead approach this setting by training a lightweight gating policy on top of a planner to select state-dependent planning budgets. Across real-time Pac-Man, Tetris, Snake, Speed Hex, and Speed Go, our gating policy outperforms fixed-budget and heuristic baselines, and transfers to a real-time setup where the environment and agent run on two different GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。