让智能体学会适时等待,减少无效决策。
Let it Cook: Learning to Wait in Sequential Decision Making

- 设计等待策略,通过延迟行动节省资源。
- 在多项任务中实现超50%时间的合理等待。
- 适合需要节能或多任务切换的场景。
在序列决策中,智能体通常每一步都感知环境并行动。但某些任务(如煮咖啡)在特定阶段无需持续干预,环境自然演化即可。此时智能体可选择等待,以节约资源或转向其他任务。本文提出训练一种‘等待策略’,决定何时、何地以及等待多久。该策略通过放弃感知来执行等待动作,表现为固定时步的主动暂停。我们以最小化感知和决策频率为目标,同时不牺牲任务性能(如任务总耗时),形式化为“学会等待”。采用分层目标的强化学习方法进行训练,在4个离散状态的家庭任务和3个连续状态环境中验证,结果表明该方法能有效学习等待行为,并使预训练策略在合适时机自动等待。不同任务允许的等待时间不同,但本方法始终能找到显著的等待方案,有时等待时间占任务总时长超过50%。
原文摘要 · Abstract (English)
In sequential decision making, an agent typically observes its environment and acts at every timestep. However, such active participation may not always be necessary; tasks such as brewing coffee include periods that are served equally well by letting the environment evolve without constant monitoring and control. During such periods, the agent could simply wait to conserve its resources, or redirect its attention to another task. We capitalize on these opportunities by training a "waiting policy" that decides where and how long to wait. This involves forgoing sensing to commit to a wait action, representing a deliberate pause for a set number of timesteps. We formalize "learning to wait" as minimizing the frequency of sensing and decision making without sacrificing task performance (e.g., the total amount of time to complete a task). To train a waiting policy, we propose an approach that employs reinforcement learning with lexicographically ordered objectives. In experiments across 4 discrete-state household tasks and 3 continuous-state environments, we show that our approach successfully learns waiting behaviors, and can adapt pre-trained policies to wait where appropriate. While different tasks permit different amounts of waiting without sacrificing task performance, our approach consistently finds solutions with significant waiting, sometimes waiting for over 50 percent of the task duration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。