让强化学习更安全：避免对特定个体造成伤害
Counterfactually Safe Reinforcement Learning
- 从反事实角度定义个体伤害，即选择动作后结果比替代方案差
- 提出两阶段算法，在提升平均收益的同时控制伤害率
- 理论证明策略在有限样本下仍能有效控制伤害，适合高风险场景
强化学习算法通常以最大化群体期望回报为目标，但平均最优的策略可能对某些个体不优，带来安全隐患。为此，本文从反事实视角形式化个体伤害概念，将伤害定义为所选动作导致的结果严格劣于基线替代方案的事件。提出一种通用的两阶段策略学习方法，在最大化期望回报的同时考虑个体伤害。进一步建立了所学策略的有限样本性质，推导出其次优性差距的上界，并证明伤害率可被良好控制。在模拟和真实数据集上的数值实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Reinforcement learning algorithms are generally designed to maximize the expected return across a population. However, a policy that is optimal on average may be suboptimal for certain individuals, leading to potential safety concerns. To address this, we first formalize the notion of individual harm from a counterfactual perspective and define harm as the event in which a chosen action results in a strictly worse outcome than a baseline alternative. We then propose a general two-stage procedure for learning policies that maximize the expected return while accounting for individual harm. We further establish the finite-sample properties of the learned policy, derive an upper bound on its sub-optimality gap, and show that the harm rate remains well-controlled. Numerical experiments on both simulated and real-world datasets demonstrate the effectiveness of the proposed approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。