让智能体向导师求助，可缓解目标误泛化问题。
Getting By Goal Misgeneralization With a Little Help From a Mentor
- 引入求助机制，让智能体在陌生情境向监督者求援。
- 求助显著提升智能体在分布外场景的性能表现。
- 基于内部状态的求助策略滞后，需针对性优化策略设计。
强化学习智能体在训练中表现良好，但在真实部署中常面临分布偏移问题。其中最严重的是目标误泛化：智能体在训练中学习到一个与真实目标重合的代理目标，但在部署时不再匹配。本文探讨了在陌生情境下允许智能体向监督者求助是否能缓解此问题。研究聚焦于在CoinRun环境中使用PPO训练的智能体，该环境已知存在目标误泛化现象。评估了多种决定何时求助的方法，发现持续求助能有效提升性能。然而，基于智能体内部状态的求助方法未能主动请求帮助，而是等到错误发生后才响应。进一步分析表明，智能体的内部状态并未表征硬币信息，凸显学习精细表征的重要性，以及忽略非即时奖励相关因素的风险，并强调需针对训练算法设计定制化的求助策略。
原文摘要 · Abstract (English)
While reinforcement learning (RL) agents often perform well during training, they can struggle with distribution shift in real-world deployments. One particularly severe risk of distribution shift is goal misgeneralization, where the agent learns a proxy goal that coincides with the true goal during training but not during deployment. In this paper, we explore whether allowing an agent to ask for help from a supervisor in unfamiliar situations can mitigate this issue. We focus on agents trained with PPO in the CoinRun environment, a setting known to exhibit goal misgeneralization. We evaluate multiple methods for determining when the agent should request help and find that asking for help consistently improves performance. However, we also find that methods based on the agent's internal state fail to proactively request help, instead waiting until mistakes have already occurred. Further investigation suggests that the agent's internal state does not represent the coin at all, highlighting the importance of learning nuanced representations, the risks of ignoring everything not immediately relevant to reward, and the necessity of developing ask-for-help strategies tailored to the agent's training algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。