通过代理价值传播，让AI高效学习人类行为并保持安全对齐。
Learning from Active Human Involvement through Proxy Value Propagation
- 设计代理价值函数，用高值标注人类示范动作，低值标记被干预动作。
- 利用TD学习将示范数据的价值传播至智能体探索的未标注数据。
- 无需修改现有算法，可适配多种控制设备和复杂任务如游戏驾驶。
主动人类参与的学习使人类在训练过程中能够主动干预并指导AI代理。人类的交互与纠正反馈提升了学习过程的安全性与对齐性。本文提出一种名为代理价值传播（Proxy Value Propagation）的无奖励主动人类参与方法，用于策略优化。核心思想是设计一个代理价值函数来表达人类意图：在人类示范中，状态-动作对被赋予高值，而被干预的智能体动作则获得低值。通过TD学习框架，示范状态-动作对的标注值被进一步传播到智能体探索生成的其他未标注数据上。该代理价值函数由此引导出忠实模仿人类行为的策略。人机协同实验表明该方法具有通用性与高效性。仅需对现有强化学习算法进行最小修改，即可在连续与离散控制任务中学习，适配多种人类控制设备，包括在《GTA V》中的复杂驾驶任务。演示视频与代码已公开于：https://metadriverse.github.io/pvp。
原文摘要 · Abstract (English)
Learning from active human involvement enables the human subject to actively intervene and demonstrate to the AI agent during training. The interaction and corrective feedback from human brings safety and AI alignment to the learning process. In this work, we propose a new reward-free active human involvement method called Proxy Value Propagation for policy optimization. Our key insight is that a proxy value function can be designed to express human intents, wherein state-action pairs in the human demonstration are labeled with high values, while those agents' actions that are intervened receive low values. Through the TD-learning framework, labeled values of demonstrated state-action pairs are further propagated to other unlabeled data generated from agents' exploration. The proxy value function thus induces a policy that faithfully emulates human behaviors. Human-in-the-loop experiments show the generality and efficiency of our method. With minimal modification to existing reinforcement learning algorithms, our method can learn to solve continuous and discrete control tasks with various human control devices, including the challenging task of driving in Grand Theft Auto V. Demo video and code are available at: https://metadriverse.github.io/pvp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。