让强化学习智能体在遵守伦理规则前提下高效完成任务
Deontically Constrained Policy Improvement in Reinforcement Learning Agents
- 用道义逻辑约束策略改进，实现任务目标与行为规范的双目标优化
- 算法能收敛到满足约束条件下的任务效用局部最优解
- 适合需要伦理合规的自动驾驶、机器人决策等场景
马尔可夫决策过程（MDP）是机器学习中处理不确定性决策的主流模型，其包含非确定性、概率不确定性及显式动作建模。强化学习（RL）代理通过最大化效用函数来学习在MDP中的行为策略。本文研究在满足道义逻辑约束条件下，最大化任务效用的策略学习问题。其中，效用函数代表任务目标（如从A快速到达B），而道义公式则表示禁止某些行为类别的伦理、社会或情境约束。本文采用期望行动功利主义逻辑（一种可解释于可控MDP的概率stit逻辑），提出一种策略改进的变体方法，证明其能收敛至任务效用的受限局部最大值。由于stit逻辑中责任源自价值最大化，该方法可视为在双层结构中同时最大化两个价值函数（一个显式，一个隐式）。实验在多个样本MDP上验证了结果的有效性。
原文摘要 · Abstract (English)
Markov Decision Processes (MDPs) are the most common model for decision making under uncertainty in the Machine Learning community. An MDP captures non-determinism, probabilistic uncertainty, and an explicit model of action. A Reinforcement Learning (RL) agent learns to act in an MDP by maximizing a utility function. This paper considers the problem of learning a decision policy that maximizes utility subject to satisfying a constraint expressed in deontic logic. In this setup, the utility captures the agent's mission - such as going quickly from A to B. The deontic formula represents (ethical, social, situational) constraints on how the agent might achieve its mission by prohibiting classes of behaviors. We use the logic of Expected Act Utilitarianism, a probabilistic stit logic that can be interpreted over controlled MDPs. We develop a variation on policy improvement, and show that it reaches a constrained local maximum of the mission utility. Given that in stit logic, an agent's duty is derived from value maximization, this can be seen as a way of acting to simultaneously maximize two value functions, one of which is implicit, in a bi-level structure. We illustrate these results with experiments on sample MDPs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。