用拍卖机制让多个目标自动协调,实时动态调整策略。
Auction-Based Online Policy Adaptation for Evolving Objectives
- 每个目标由独立策略竞争出价,高价者执行动作。
- 动态增删目标时,只需增减对应策略,无需重新训练。
- 适合目标频繁变化的多任务强化学习场景。
我们研究多目标强化学习问题,其中目标属于同一类(如可达性目标),且在运行时可能动态出现或消失。目标是设计可自适应调整行为的策略。为此,我们提出一种模块化框架:每个目标由一个自私的本地策略支持,通过新颖的拍卖机制实现协调——策略根据当前状态紧迫性出价,最高出价者选择动作,从而实现目标间的动态、可解释权衡。当目标变化时,系统仅需增删对应策略即可适应。由于目标来自同一家族,可部署相同参数化的策略副本,实现实时运行时快速适应。我们将问题转化为广义和博弈,使各策略在竞争中优化自身目标的同时,学会生成反映相对优先级的合理出价。在温和假设下,我们证明了纳什均衡存在:不诚实出价会导致次优结果,最紧迫的目标能自动获得控制权。实验中,策略采用近端策略优化(PPO)并发训练,在两个Atari游戏和一个基于网格的世界路径规划任务(含动态目标)上,性能显著优于单体式PPO策略。
原文摘要 · Abstract (English)
We consider multi-objective reinforcement learning problems where objectives come from an identical family -- such as the class of reachability objectives -- and may appear or disappear at runtime. Our goal is to design adaptive policies that can efficiently adjust their behaviors as the set of active objectives changes. To solve this problem, we propose a modular framework where each objective is supported by a selfish local policy, and coordination is achieved through a novel auction-based mechanism: policies bid for the right to execute their actions, with bids reflecting the urgency of the current state. The highest bidder selects the action, enabling a dynamic and interpretable trade-off among objectives. Going back to the original adaptation problem, when objectives change, the system adapts by simply adding or removing the corresponding policies. Moreover, as objectives arise from the same family, identical copies of a parameterized policy can be deployed, facilitating immediate adaptation at runtime. We show how the selfish local policies can be computed by turning the problem into a general-sum Markov game, where the policies compete against each other to fulfill their own objectives. To succeed, each policy must not only optimize its own objective, but also reason about the presence of other goals and learn to produce calibrated bids that reflect relative priority. Under mild assumptions, we prove the existence of Nash equilibria where dishonest bidding leads to suboptimal outcome, and the most urgent objectives win control automatically. In our implementation, the policies are trained concurrently using proximal policy optimization (PPO). We evaluate on two Atari games and a gridworld-based path-planning task with dynamic targets. Our method achieves substantially better performance than monolithic policies trained with PPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。